The demo went well: five questions, five fluent answers, each with a reference to a document. The doubts come later. Were the answers right? Did they come from the current version of the procedure? What if the person asking did not have access to that document? A well-written answer proves none of this. Evaluating an assistant that answers from documents takes a method, and that method needs to exist before the first demo.
RAG in plain language
Most of these assistants work in two steps, in a design called RAG (retrieval-augmented generation):
- Retrieval. The question is used to search an index of authorized documents for the most relevant passages.
- Generation. A language model receives the question and those passages, and writes the answer based on them, pointing to where each statement came from.
The model does not “know” the company’s documents: it reads whatever the search returned at that moment. That means there are two different points of failure. The search can bring back the wrong passage, or nothing useful. And the model can write something the passages do not say. A good evaluation measures the two separately, because each one is fixed differently.
Reference questions, validated by experts
The core of the evaluation is a set of reference questions. For each question, record:
- the question the way a real user would ask it, with the usual abbreviations and vagueness;
- the correct answer, written or approved by a subject-matter expert;
- the document and passage that support that answer;
- the access profile of the person asking.
Deliberately include questions the documents cannot answer. For those, the right behavior is to say no supporting evidence was found, not to fill the gap with the model’s general knowledge. An assistant that never says “I don’t know” is hiding a problem.
Split the set in two: one part for tuning the solution during development and another held back for acceptance testing. When the same questions are used to tune and to approve, the final score comes out optimistic.
Does the cited source support the answer?
Citing a source is not the same as being supported by it. For each answer, the reviewer checks three things:
- Is the answer correct compared with the reference answer?
- Did the search return the right passage? If the passage that supports the answer was not retrieved, the problem is in the index or the search, not the model.
- Does the cited passage say what the answer claims? Answers that add details missing from the source, or that cite a real document for a claim it does not contain, count as errors, even if the claim happens to be true.
Superseded documents and revoked access
Two tests are often left out, and they are the ones that matter most in production.
Superseded document. When a procedure gets a new version, the old one has to leave the index or be marked as retired. Include questions whose answer changed between versions and confirm the assistant answers from the current one. Then publish a new version of a test document and measure how long it takes for the assistant to reflect the change.
Revoked access. The assistant must answer each person only with what that person is allowed to see. Test with at least two profiles: the same set of questions, asked by someone who has access to a document and by someone who does not, should produce different answers. Then remove a test user’s access and confirm the content stops appearing for them. Search services such as Azure AI Search offer document-level permission filters, but the configuration and the syncing of those permissions have to be tested in the real environment. A single case of improper access should block the release.
Model-as-judge: support, not a substitute
Reviewing hundreds of answers by hand is expensive, which is why there are automated evaluators in which one language model scores another: whether the answer is supported by the passages, whether it is relevant, whether it is complete. They are useful for rerunning the evaluation after every change to the model, the instructions or the index, and for quickly surfacing suspicious cases.
But the judge makes mistakes too, and it can make them systematically. Use it this way:
- calibrate it by comparing the judge’s scores with expert scores on a sample of the same answers;
- record the version of the judge model and its instructions, because the score changes when they do;
- keep the acceptance decision with the experts, supported by the judge’s numbers.
Cost per accepted answer
The cost of running an assistant is not just model usage. Over a given period, add up:
- model and search usage;
- infrastructure and index storage;
- the time people spend reviewing and correcting answers.
Divide by the number of answers the reviewers accepted, not by the total number of answers. An assistant that answers a lot and gets little right has a high cost per accepted answer, even with low usage costs. That is the number to compare against the baseline of the current process.
Report sample size alongside rates
“92% accuracy” says little without the sample size. As a worked example: 46 correct answers out of 50 questions is 92%, but the 95% confidence interval (Wilson method) runs from about 81% to 97%. With 500 questions and the same proportion, the interval is much narrower. So:
- always report “46 of 50,” not just “92%”;
- break the rates down by question type (answerable, unanswerable, by access profile), because the overall average hides the category that fails;
- rerun the evaluation on the same set after every relevant change and compare it with the previous round.
With this method, the decision to expand, adjust or end a pilot rests on recorded evidence. Before you get here, it is worth checking whether your company’s data is ready for AI for the task you chose.
Want to assess a use case?
If your company is testing an assistant on internal documents, or thinking about starting, an assessment helps define the reference questions, access profiles and acceptance criteria. Learn about our enterprise data and AI service or request an assessment.