Ask a new assistant three questions in a meeting and it may feel remarkable. Ask it the same questions fifty times, with missing records, ambiguous names, changed documents and a slow network, and you discover the product you are actually preparing to ship. The difference is evaluation: a repeatable way to check whether the system does the useful thing for the right people.
The easiest test is the least informative
Start by defining the user task, not a generic model benchmark. For a document assistant, success might mean finding the correct policy and linking it; for an extraction feature, it might mean producing valid fields that a person can correct. Write down the unacceptable outcomes too: citing a document the user cannot access, creating a duplicate transaction or hiding uncertainty.
- Define one measurable user task.
- Collect ordinary, difficult and missing-information examples.
- Decide which errors are tolerable and which should block release.
Separate the stages you can fix
If the feature uses retrieval, test whether the necessary passage was found before grading the generated answer. If it calls tools, inspect whether it chose the right tool with valid arguments before judging the final message. This separation tells you whether to change the index, the tool contract, the prompt, the model or the interface.
For each case, record the input, allowed context, expected behavior, actual behavior and a short reason for the score. A single numeric quality score is helpful for tracking progress but poor at explaining failure. Review a sample by hand; automated judges can help triage a larger set but should themselves be checked against human decisions. Microsoft’s RAG guidance explicitly separates retrieval and end-to-end evaluation.
- A failure to retrieve is different from a failure to interpret.
- A correct answer from forbidden context is still a failure.
- A safe refusal to answer can be the correct outcome.
Test the experience around the model
Measure the time a user waits for the first useful response and the time to finish the task. Record cost per completed task, not just cost per API call. Validate accessibility of streamed updates, meaningful loading states and the ability to undo or correct a response. People need a legible failure state when an integration is unavailable.
Before wider release, set a small rollout group and watch actual examples with consent and appropriate redaction. Track corrections, escalation to a human, repeated questions and drop-off. A test set will never anticipate every real-world request; observation after launch is how the test set improves.
An evaluation case worth keeping
Try this test: the source document says a benefit applies to employees in one country. Ask the assistant whether an employee in another country is eligible. A fluent answer that borrows the same words from the source should fail if it ignores the region. A refusal with a link to the policy owner may pass. This is why a test set needs cases that exercise conditions, not only examples with easy verbatim answers.
Write the expected behavior before running the model. Otherwise people tend to rationalize a plausible output after seeing it. Keep a small review log with the exact document version, model configuration and observed output. That log becomes valuable when a later document update changes the result.
- Include a condition or exception.
- Specify what the assistant should do if the answer is unknown.
- Rerun the case after any retrieval or model change.
Release with a change loop
Every material model, prompt, retrieval or tool change should run against a versioned test set. Keep examples of past incidents in that set so a fix for one problem does not quietly restore another. Document the owner of the feature and the route for reporting a harmful or incorrect result.
When the feature consistently solves its bounded task, behaves sensibly outside that task and gives people a clear way back to control, it is worth shipping. “The model seems smart” is a starting observation; the product standard is more demanding.