Imagine an employee asking whether their team can reimburse an international train ticket. A plausible answer is easy for a language model to write. A correct answer depends on the current travel policy, the employee’s location, the ticket date and perhaps an exception approved elsewhere. That difference is the reason to build retrieval augmented generation, or RAG.
Start with a question, not a vector database
RAG combines a search step with a generation step: find relevant material, then write a response grounded in it. The difficult work is deciding what material belongs in the search index, how it should be split and who may see it. A model cannot rescue a missing or unauthorized source. Microsoft’s RAG design guidance treats preparation, retrieval and evaluation as distinct stages for good reason.
- Collect real questions and locate the passages that answer them.
- Assign owners and update dates to source documents.
- Exclude drafts, duplicates and material users should never access.
Retrieval should preserve meaning and access
Splitting a policy into arbitrary fixed-size pieces can detach a rule from its exception. Start with semantic boundaries such as headings, sections and tables. Keep source title, version, page and access scope as metadata. At query time, filter by the viewer’s permissions before ranking candidate passages; filtering after generation is too late.
Create a small benchmark with answerable questions, near misses and questions whose answers are absent. For each, record the passages a capable employee would need. Now you can test search separately from the model: did the correct passage appear, at a useful rank, for the right user? If not, improve the index or the query strategy before tuning the prose of the answer.
- Test a document with an exception far from the main rule.
- Test users with different access to the same knowledge base.
- Test a newly updated policy alongside an outdated duplicate.
Make uncertainty visible to the reader
A good response says what it found, links the supporting passages and distinguishes a sourced answer from an inference. When the documents do not settle the question, the assistant should say so plainly and offer the next sensible route: the policy owner, a form or a human reviewer. “I could not find a rule for this journey” is more useful than a confident guess.
Look beyond whether the final sentence sounds right. Microsoft’s RAG evaluators distinguish retrieval quality from response groundedness. A system can retrieve the correct passage and still misread it; it can also produce a sensible sounding answer after retrieving the wrong document. Keep examples of both failure modes in your evaluation set.
What to show in the interface
For the travel-policy example, place a short answer at the top. Immediately below it, show the policy title, the relevant excerpt and the policy date. If the policy says “domestic rail travel” and the question is international, the assistant should explain that the available policy does not settle the case. A link labelled “Ask the travel team” is more useful than a longer speculative answer.
Now test the layout with a reader who has access only to one region’s policies. If the interface reveals the title of a restricted policy while hiding the passage, the system has still leaked information. Treat titles and snippets as protected content when necessary. The product experience and the permission model must agree.
- Keep the citation beside the claim it supports.
- Show version and date when policy freshness matters.
- Make an unresolved question easy to hand to a person.
Ship an answer people can verify
Present citations next to the claims they support, not in a decorative list hidden at the bottom. Provide the document date when freshness matters. Add a lightweight way for readers to flag a broken citation or outdated answer, and send that feedback to the person responsible for the source.
The best knowledge assistant makes the organization’s information easier to use while preserving the reason it is trusted. It does not replace stewardship of the underlying documents. The operational plan should include reindexing, permission changes and a way to withdraw a bad source as quickly as it was added.