We measured what AI can really do with the documentation of a real IT project (part 1)

Published: 24 min read
We measured what AI can really do with the documentation of a real IT project (part 1)

Part one of a two-part series: setting the stage and the goals

Anyone who has worked on a large IT project knows the situation: a new colleague is looking for one specific piece of information in the material of a project spanning several years, where the requirement is in the requirements specification, the related test cases are in a test report, a year passed between the two and, in the best case, several properly versioned copies were uploaded to different folder structures. When an auditor asks during a review who actually tested the logging of failed login attempts, someone has to sit down and leaf through hundreds or thousands of pages, often of material they neither wrote nor know. Today it would be an obvious choice to use a language model for this work: these systems answer in moments, and their answers usually sound convincing. But a convincing answer and a correct answer are not the same thing.

With new expectations and promises about AI appearing practically every month, and with large language models increasingly being used without control, we decided to measure what these systems can do on the kind of documents we work with every day. We took the complete Hungarian-language document estate of a completed, multi-year energy-sector development project, forty-seven documents in all, containing many thousands of pages of technical and methodological documentation, feasibility studies, requirements specifications, test reports and acceptance criteria, and on this basis we put six hundred and seventy questions to five different large language models. In this article we set out to show, first, why we were interested in solutions that can run on our own servers, what we wanted to use the models for, and how we put the measurement together so that the result can be trusted. The results of the test are covered in depth in the second article of the series.

What existing benchmarks show, and what they leave unanswered

Many studies have tried to address the subject of this paper, but in many cases they do not capture the problem that matters to us. The most widely used international benchmarks are built on English-language Wikipedia articles, such as HotpotQA and MuSiQue [1, 2]; others work from news articles (MultiHop-RAG [3]) or from the financial filings of US listed companies (FinanceBench [4]). The debate about the effectiveness of solutions that search along the table of contents is still under way, and the results contradict each other: the developers of PageIndex reported 98.7 percent accuracy for their PageIndex-based system [5], while researchers at PwC, on a similar benchmark of their own built from US stock-exchange filings, found that vector search directed by the model gave the better answer in 68 percent of cases [6]. Hungarian research centres, among them the Hungarian Research Centre for Linguistics, Eötvös Loránd University (ELTE) and the artificial intelligence research group in Szeged, mainly study how well the models handle Hungarian and how text should be prepared for retrieval. We did not, however, find a study that examines as a whole the deliverables of the full life cycle of a real project spanning both IT and the energy sector: a document estate whose parts follow from and build on one another, and where a good answer requires finding, interpreting and comparing these complex professional materials at the same time. We therefore carried out the research ourselves, from data preparation through measurement to evaluation, with several models and several kinds of retrieval.

Why we examined solutions that run in our own environment

With a cloud service, the question and the business information, and with them the relevant part of the document, leave our own environment; the cost can grow significantly with use; and the provider can replace the model at any time, in which case the same question may receive a different answer the next day: instructions tuned to the old model work differently, the accuracy measured earlier no longer holds, and reproducibility may suffer. If the system runs on our own server, the document stays in our own secure business environment, every step can be logged, the cost is a one-off investment, and the model stays the same until we decide otherwise: we move to a new version only after we have measured it again on our own question set, and the old one can be kept for re-checking earlier answers. A solution running on our own server does not guarantee a word-for-word identical answer either, because there is randomness in the way language models work: in our own measurement, the same configuration run twice gave an almost identical result overall, but individual questions did sometimes receive different answers. What running the system ourselves gives us is traceability: we know which model version answered, with what settings and on the basis of which passages, and the same environment can be rebuilt later. This naturally has a price: a server has to be bought and operated, and several of today’s strongest models cannot be installed on it at all, or only at a very substantial one-off capital cost. Only so-called open-weight models can be installed in one’s own environment: models whose trained parameters, that is, their weights, have been published by the developer, so that the model can be downloaded and run by the organisation on its own machine. Closed models, which include most of the best-known frontier models, can be accessed only through the developer’s service.

What regulation adds

At present no legislation explicitly requires artificial intelligence to be run only on one’s own machine or in one’s own environment, but regulation places a growing burden of proof and control on organisations working with sensitive data, and running the system themselves allows them to exercise that control directly. In Hungary, the Cybersecurity Act (Act LXIX of 2024, Section 9) and its implementing decree (Government Decree 418/2024 (XII. 23.)) make it dependent on data classification whether a defined group of the organisations concerned may use a non-private cloud service or process data abroad. A non-private cloud is one where the provider serves many customers on the same infrastructure; the examples given by the cybersecurity authority are the generally available services of Microsoft 365, AWS, Azure and Google Cloud, although the same providers also offer segregated, private solutions. In a private cloud, by contrast, all hardware and software is at the organisation’s disposal and can be accessed only from within the organisation [7, 8]. The decree distinguishes four classes (F1–F4), and classification depends on how much damage would be caused if the data fell into unauthorised hands, were corrupted or became unavailable. As examples of the level involving significant damage, the explanatory memorandum to the decree mentions precisely project documents, organisational policies and tenders, and at the critical level business secrets and special-category or large-scale personal data. As a general rule, data in the highest class (F4) may be processed only within the territory of Hungary, and in the cloud only in a private or government cloud certified by a third party. Anyone who would use a non-private cloud or process data abroad must, before the decision, prepare an exit plan showing how they would return to a local environment or a private cloud; some of the organisations concerned must also demonstrate with a cost-benefit analysis that the cloud offers well-founded advantages (Section 3). In the financial sector, the EU regulation on digital operational resilience (DORA, Regulation (EU) 2022/2554) has demanded similar discipline since 17 January 2025: dependence on external ICT providers must be managed, and services supporting critical or important functions require a documented and tested exit strategy.

EU legislation is moving in the same direction, as both the laws already adopted and those under negotiation show. The Artificial Intelligence Act (Regulation (EU) 2024/1689) requires logging, human oversight and detailed documentation for high-risk systems, placing different responsibilities on the provider and on the organisation deploying the system; the amending regulation that entered into force in July 2026 (Regulation (EU) 2026/1744) postponed the application of these obligations to 2 December 2027 for systems under Annex III and to 2 August 2028 for those linked to Annex I, so there is time to prepare, but the requirement is there. The regulation does not say where the system should run, but it does say that its operation must be logged, documented and supervised, wherever it runs. On 3 June 2026, as part of its technological sovereignty package, the European Commission also submitted its proposal for a Cloud and AI Development Act (COM(2026) 502), which would introduce four sovereignty levels for public-sector cloud use and require Member States to carry out sovereignty risk assessments; according to the draft, this could later be extended to private operators in the sectors covered by cybersecurity regulation [9]. The proposal is not yet law, but it signals that control over cloud services and data sovereignty are becoming a regulatory issue in their own right. Anyone working with sensitive data and documents will therefore increasingly have to show where their data is, who has access to it, and how they could change provider. When the organisation runs the system itself, these questions are easier to answer. If, for example, an internal auditor or the authority asks where the contracts uploaded to the system, the questions asked about them and the search indexes are stored, the answer is a single sentence: on a server in the organisation’s own server room, accessible to staff designated under our own access rules; every query appears in our own log, and the organisation itself decides on retention periods and logging rules. With a cloud service, the same answer requires checking the location of data processing, the sub-processors used, the rules on access and data transfer, and the conditions for deleting and retrieving data, while changing provider involves data migration and re-validating the system. These risks can be reduced with a suitable cloud architecture and contractual guarantees, but in this the organisation depends on the provider’s cooperation.

What the market shows

Market data show the same demand, but with more nuance than is often heard. According to Gartner’s forecast, European organisations’ spending on sovereign cloud will more than triple between 2025 and 2027, from 6.9 billion to 23.1 billion dollars [10]. In a representative survey by the German industry association Bitkom, 85 percent of companies said the country is too dependent on US cloud providers, and 64 percent of companies using the cloud feel that US government policy forces them to rethink their cloud strategy [11]. According to a vendor-commissioned survey of 1,800 IT leaders, the share of companies running their production AI systems primarily in the public cloud fell from 56 to 41 percent in one year [12]. An independent IDC survey, however, found an even split between private and public environments [13], and according to Bitkom’s data the use of AI services delivered from the cloud also continues to grow. Regulation and the market are therefore moving not towards a single solution, but towards greater control. One form of this is the sovereign cloud; the other is one’s own environment, which is the most obvious choice where the data is most sensitive. That this is an option at all is due to the progress of open-weight models: according to Epoch AI’s measurements, the best of them trail the strongest closed models by four months on average [14].

The business use cases we had in mind

We looked for a solution to three situations, all familiar from our daily work.

The first is when someone expects a precise, traceable answer from a single document. An auditor, a quality assurance specialist or a business analyst is not satisfied with being told that, according to the system, the contract requires a daily backup; they want to see in which clause and under what conditions. Here an almost-correct answer can be worse than a wrong one, because it is less likely to be noticed. An obviously wrong answer can be spotted and discarded by the analyst. But if the system quotes the backup requirement correctly, only from an earlier, since-amended version of the contract, the answer is plausible and the source reference appears to be in order.

The second is when several documents have to be placed side by side. Has what the requirement prescribes been fulfilled, is there a test for it, and does the test plan say something different from the test report? Quality assurance and acceptance consist largely of such questions. Today this is done by people, with a great deal of manual work and checking by hand. In our view, this area may be one of the most important untapped business opportunities for the responsible use of artificial intelligence.

The third is targeted search in the knowledge held in a given document estate: a new colleague or a customer service agent asks which policy covers remote access, and does not want to wait two minutes for the answer. The questions here are simpler, but speed is a requirement.

The questions we asked

We broke the first two situations down into question types, because the statement that a system “answers from documents” is too broad to be measured. We did not choose the question types at random. We started from where smaller models tend to fail: they stop at the first document when the search should continue in a second; they claim coverage where there is none, because “there is a test for it” is linguistically a more probable answer than “there is none”; and they nod along to a plausible-sounding statement even when the source says otherwise. The table lists all fifteen types with an example of each.

Question typeWhat the system has to be able to doExample
Within one document (351 questions)
Finding a specific factFind a piece of data where it is written downHow many days does the trial operation last?
Assembling an answer from several sectionsPut the answer together from parts that are far apartHow does the data intake module connect to the data processing module, and what error handling applies to it?
Following an internal referenceGet to where the text pointsSection 3.2 refers to Annex 7. What does it require there about backups?
Giving the exact locationSay where something appearsWhich chapter describes the authorisation levels?
Detecting missing contentNotice that something is not in the document estateDoes the installation guide include the restore procedure? (The expected answer is not a bare “no”: it must also explain what the document covers instead.)
Recognising an unanswerable questionState that the documents give no answer to thisWhat is the system’s annual operating cost?
Two documents together, with an explicit link: the documents themselves indicate that they belong together
Matching a requirement with its evidenceOne document prescribes, the other provesThe contract requires a daily backup. Does the test report prove that it was implemented?
Checking that content is missingState that the evidence is missingDoes the test report contain anything about round-the-clock on-call support?
Checking for a contradictionCompare two statementsOne plan gives 300,000 endpoints, the other 500,000. Do they contradict each other?
Following a reference into the other documentGo to the referenced document and find the answer thereThe specification refers to Section 4.2 of the architecture plan. What does that section record?
Identifying copied textRecognise that word-for-word repetition is not yet evidenceThe report repeats the requirement word for word but gives no result. Does it prove fulfilment?
Comparing versionsSay what has changedHow does the phase three requirements specification differ from that of phase two?
Two documents together, with an implicit link: related in content, without a reference
Link through a shared codeNotice that the same identifier appears in two placesA requirement code appears in both the specification and the test, but neither refers to the other.
Link through a shared termRecognise the same technical term, without a codeBoth documents write about single sign-on.
Same content in different wordingFind the same idea behind different wordsOne prescribes “logging of failed login attempts”, the other records: “wrong password three times, log entry created”.

The 319 questions involving two documents fall into two groups, and this distinction was one of the most important decisions of the measurement. In the first six (explicit link), the documents themselves indicate that they belong together, for example by referring to each other. In the last three (implicit link) there is no such handhold: the two documents are about the same thing, or their content together answers a more complex question, but neither refers to the other. In practice this is quite a common case, and how well the models handle tasks of this kind has a significant effect on their overall performance.

The documents

For the measurement we used the complete documentation of a development project delivered in several phases: forty-seven documents, roughly 4,700 pages in total. They include requirements specifications, an architecture and infrastructure plan, a programmer’s manual, a test plan, test and trial-operation reports, a defect ticket summary, supplementary submissions and an installation guide. These documents build on one another: one prescribes something, another designs it, a third proves that it was done. Several of them appeared in successive versions, so the system also has to find the right version within the right document. We do not name the project or the client, and the examples in the table are illustrations modelled on the real questions.

The models and the retrieval

We tried five large language models. Four of them are open-weight and can therefore be installed on own servers. The smallest, a 26-billion-parameter model, runs on a desktop-sized server with 128 gigabytes of memory; the three larger ones, between 180 and 320 billion parameters, need a professional graphics card; the fifth is a closed model available only as a service, which we included as a reference point (GPT-5.6 Luna). (The parameter count indicates the size of the model: the more parameters a model has, the more memory it needs to run. This is why the smallest fits on a desktop machine and the larger ones do not.) We ran the smaller model (Gemma 4, the 26-billion-parameter version) on our own server, an NVIDIA DGX Spark, and the larger models (Qwen3.8-Flash, DeepSeek V4-Flash, GLM-5.3-Flash) from the cloud in this round; we point this out again with the results in part two.

Every model has to search the documents before it answers, and it could do so in three ways. The first two are built on vector search, meaning that the system retrieves passages similar in meaning to the question. In the simplest version (RAG workflow), the system searches once, passes the eight passages judged best to the model, and the model has to answer from those. In the second version, which the field calls Agentic RAG, the model itself directs the search: if it has not found what it wanted, it rephrases the question, narrows it to one document, or follows a reference. The third, structural navigation, follows the principle of a solution called PageIndex, in a simplified form [5], and relies on the documents’ tables of contents instead of vectors. It works the way a person would: it looks at the tables of contents, selects the documents and sections that may be relevant, and reads those.

The steps and typical response time of the three search modes tested

Figure 1. The three search modes compared in the measurement. The feedback loop of Agentic RAG means that if the passages found are not enough for an answer, the model rephrases the query, narrows it to one document or section, or follows a reference, and searches again, in up to fifteen steps. The bar on the right shows how long an answer typically took to arrive: half of the measured answers were faster than this, half slower.

How the links became questions

Putting the questions together took more work than the measurement itself. The hardest part was the questions that require knowing two documents at the same time, because for these we first had to find out which documents belong together at all. With forty-seven documents it is not realistic to do this entirely by hand, so we had a program find the links: where one document refers to another, where the same requirement identifier appears in both, where the same rare technical term is used. After several rounds of narrowing down, 204 document pairs remained for which we could prove that they belong together.

A question can be used for measurement only if we record in advance what the correct answer is and provide the evidence that goes with it. The field calls this the ground truth. In our case, the ground truth of every question consists of four parts:

  • the expected answer in full sentences,
  • the list of key facts that must appear in a good answer,
  • the logic of the answer, that is, the reasoning showing how the answer follows from the sources, and
  • the verbatim source quotes showing which sentences of the document it is based on.

For a question about something missing, for example, the expected answer is not simply “no”, but also what the requirement prescribes and what the other document covers instead. Recording the sentences that support an answer is common practice in international benchmarks as well, for example in HotpotQA [1]; we tightened this by requiring the quote to appear character for character in the source, which is checked by a program. In scoring, this makes it possible to judge not only whether the conclusion of an answer is right, but also whether it is right for the right reason.

We had the questions for these pairs written by the strongest model available at the time of the research, together with the expected answer and two verbatim quotes, one from each of the two documents. We called the model through the API, separately for each question and with no conversation history, so that for no question could it rely on what it had seen or written for an earlier one, and no recurring pattern could form across the questions. We followed the same rule later with the answering models and in scoring: every question and every answer started from a blank slate. Then, in a second, independent pass and without knowledge of the answer, we had it checked whether the question can be answered from the given sources alone, whether both documents are really needed, and whether the wording of the question gives the solution away. We also filtered out pairs in which the two quotes were identical character for character, because that is a copy, not a link. Meanwhile a simple program checked whether the quotes could be found character for character in the source. This last check caught the most errors, because the model tends to quote almost exactly. What remained was reviewed row by row, in three rounds of corrections, by our business analyst colleagues, who know the documents, and they decided whether each question could stay. We later also examined how consistently they themselves judge; this is covered in detail in the section on scoring. 319 questions passed all of the filters above:

How the two-document questions were built, from candidate links to the final set

Figure 2. The path of the two-document questions from the links found by program to the final set. Because of the two verbatim quotes, it can be checked for every question whether the system really reached both documents needed for the answer.

Evaluating the results: scoring

It is not practical to go through several thousand answers by hand, so we entrusted scoring to a language model as well, but not to the one that gave the answer: if a model scores its own answer, it tends to let itself off. The judge was a strong external model that did not know which model and which search mode had produced the answer it was looking at; it received every answer separately, in a call with no history and with randomness switched off so that the scoring could be repeated; it worked to a rule fixed in advance; and it had to justify its verdict. It could choose from three grades: zero points for a wrong or missing answer, one for a partly correct answer, two for a correct one. A three-point scale is an established approach: Meta’s CRAG benchmark also scores answers in three outcomes (correct, missing, incorrect) and, to avoid self-preference, uses two different judge models [15]; a recent study comparing three scales found the one with least granularity, zero to five, to align most closely with human judgement [16]. Since in our case a wrong answer and a missing answer both score zero, we count abstention and fabricated content as separate indicators.

Take the question “Does the test plan prove the requirement in the technical specification concerning decision support?” If one model said this could not be determined from the documents, although the answer was in them, we gave zero points. If another described the role of the two documents correctly but worked from a version of the test plan other than the one named in the question, and decisive details were left out, we gave one point. If a third found both documents, read the right version and answered with the appropriate source, it earned two points.

We verified the judge’s reliability in two ways. We had fifteen hundred answers scored by a model from another vendor as well, in such a way that it did not know which model and which search mode had produced the answer: it received the items in random order, identified only by a serial number. Four of our business analyst colleagues, after sampling, also scored part of the answers by hand under the same rule, likewise without knowing the model or the search mode. For the sample we selected answers from both vector search and PageIndex-style structural navigation, short and long answers, answers that took a position and answers that abstained, so we could also see whether the judge favours one of the search modes.

In examining the results we looked at how often one evaluator gives zero and another gives two to the same answer, because we identified this as a significant discrepancy. Between the machine judge and the humans this occurred for 4.8 percent of answers, between two humans for 4.9 percent. The machine therefore differs from a human roughly as much as humans differ from one another, and all six evaluators saw the ranking of the model–retrieval pairings in essentially the same way. In judging partly correct answers, however, humans are not consistent either, so we draw no conclusions from small differences between scores. This is also in line with the recommendation in the literature: models should be compared on the same questions, on the basis of paired differences, and a difference within measurement noise does not count as a difference [17]. The threshold is our own rule of thumb: since the same configuration run twice differed by a few hundredths of a point, we do not treat a difference of less than one tenth as large enough to rank on.

Much of the method is established practice: the ground truth fixed in advance, the machine judge with human verification, the three-point scale and the paired comparison. Our own contribution is the grouping of the nine link types into explicit and implicit, the selection that ensures coverage, the double quote checked character for character and the human sample balanced across several criteria, as well as measuring the machine judge against the agreement between humans, as a recent international study also does [18].

The next part covers what the results of our measurement showed: which model achieved what with which search mode on each question type, what a desktop-sized server can solve reliably, what limits follow from model size, and which are the use cases where, with a good choice of retrieval architecture and model, we can successfully support our business processes with artificial intelligence.

References

[1] Yang, Z. et al. (2018): HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. arXiv:1809.09600.

[2] Trivedi, H. et al. (2022): MuSiQue: Multihop Questions via Single-hop Question Composition. arXiv:2108.00573.

[3] Tang, Y., Yang, Y. (2024): MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries. arXiv:2401.15391.

[4] Islam, P. et al. (2023): FinanceBench: A New Benchmark for Financial Question Answering. arXiv:2311.11944.

[5] VectifyAI: PageIndex. github.com/VectifyAI/PageIndex.

[6] Lumer, E. et al. (2026): Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models. ICAART 2026. arXiv:2511.18177.

[7] Act LXIX of 2024 on the Cybersecurity of Hungary (njt.jog.gov.hu/jogszabaly/2024-69-00-00); Government Decree 418/2024 (XII. 23.), Annex 1 (data classification), and its explanatory memorandum (njt.jog.gov.hu/jogszabaly/2024-418-20-22).

[8] National Cyber Security Institute (Hungary): Cloud; Questions and answers on security classification. nki.gov.hu/hatosag/tartalom/felho.

[9] European Commission: Proposal for a Regulation establishing a framework of measures for strengthening Europe’s cloud and AI ecosystem (Cloud and AI Development Act), COM(2026) 502 final, 3 June 2026. Other EU regulations mentioned in the text: (EU) 2024/1689, (EU) 2026/1744, (EU) 2022/2554.

[10] Gartner (9 February 2026): Gartner Says Worldwide Sovereign Cloud IaaS Spending Will Total $80 Billion in 2026. gartner.com/en/newsroom.

[11] Bitkom (17 June 2026): Cloud Report 2026. Representative survey of 603 companies in Germany. bitkom.org.

[12] Broadcom (2026): Private Cloud Outlook 2026. Survey of 1,800 IT decision-makers. broadcom.com.

[13] IDC survey of 304 decision-makers (December 2025); cited in TechTarget (2026): Enterprises make strides with private AI on-premises.

[14] Epoch AI (29 May 2026): Open models lag state-of-the-art closed models by 4 months. epoch.ai/data-insights/open-closed-eci-gap.

[15] Yang, X. et al. (2024): CRAG — Comprehensive RAG Benchmark. arXiv:2406.04744.

[16] Li, W. et al. (2026): Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0–5 Grading Scale. arXiv:2601.03444.

[17] Miller, E. (2024): Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640.

[18] Han, S. et al. (2025): Judge’s Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement. arXiv:2510.09738.

Try the AI Act risk classifier

Find out in 5 minutes which risk category your AI system falls into — free, 100% private, with a detailed PDF result.

Start the assessment
H
Varga Ákos
Harvey's · AI & compliance team
Experts in AI Act compliance, testing and security audits. Reach out any time with questions.
LinkedIn

Harvey's newsletter

Stay up to date with our AI Act content

One practical monthly summary on EU AI Act compliance — no spam, unsubscribe any time.

← Back to the blog