An operations team can have years of procedures scattered across shared drives, a wiki and email. A new colleague needs a way to ask a question and find the current answer without interrupting everyone else.
That's retrieval-augmented generation, or RAG: find the passages relevant to a question, then have a language model answer using those passages. The idea fits in one sentence. Getting people to trust the answers takes more work.
Retrieval decides the quality
If the search step finds the wrong passages, the best model in the world will write a confident, wrong answer. We spend more time on retrieval than on prompts.
Clean the sources first. Outdated procedures produce outdated answers. We work with the client to archive superseded documents and mark the current version of each policy. This is unglamorous and it matters more than any technical choice.
Chunk by meaning. We split documents at headings and sections rather than every N characters, and keep the document title and section path with each chunk. A passage that says "the limit is 50,000" is useless without knowing which limit.
Use hybrid search. Vector search finds passages with similar meaning. Keyword search finds exact terms such as product codes, form numbers and names. We run both and merge the results, then rerank the top candidates.
Respect permissions. Each chunk carries the access rules of its source document. A search by someone in sales never retrieves HR files, because we filter before the model sees anything.
Answers people can check
Every answer cites its sources, with links that open the document at the right section. Staff learn quickly whether to trust the tool when they can click through and see the passage.
We instruct the model to answer only from the retrieved text and to say so when the text doesn't contain the answer. "I couldn't find this in the documents. The closest match is X" is a good answer. A made-up procedure is a dangerous one.
Measure retrieval and answers separately
We build a test set with the client: a few hundred real questions with the document and section that answers each one. Then we measure two things:
- Retrieval: did the correct passage appear in the top results?
- Answer: is the answer correct, complete and supported by the cited passage?
Separating them tells you what to fix. Low retrieval scores mean chunking, search or source problems. Good retrieval with bad answers points at the prompt or model.
Keep it current
Documents change. We re-index on a schedule and when files change, remove deleted documents from the index, and show the "last updated" date of each source in the answer. Users also get a thumbs-down button that sends the question, answer and sources to a review queue.
What success looked like
The useful test is an uncommon question about a process such as refunds. If the tool finds the right steps and links to the current policy, someone can check the answer. That builds trust faster than a polished demo.
