RAG in Production: What Actually Breaks

A demo RAG system takes an afternoon. A RAG system that people rely on for real answers takes considerably longer, and almost none of the extra work is about the model.

Retrieval is the product

If the retrieved context does not contain the answer, no amount of prompt engineering will produce one — the model will simply produce something plausible instead. That failure mode is worse than an error message, because it is confident and it is wrong.

Most of the quality gains came from unglamorous retrieval work: chunking documents along semantic boundaries rather than fixed token counts, keeping enough surrounding context that a chunk is interpretable on its own, and combining keyword and vector search so exact identifiers still match.

Multi-source means multi-format

Internal knowledge does not live in one place. It is in documents, spreadsheets, ticket threads, and the heads of four people. Every source you add brings its own extraction problem, and a badly parsed table can poison answers for an entire topic area.

Normalising at ingestion — into a consistent representation with source, recency and access metadata attached — is what makes the rest tractable.

• Chunk on meaning, not on a fixed token count
• Hybrid keyword + vector retrieval so identifiers still match exactly
• Carry source and recency metadata through to the answer
• Treat access control as a retrieval filter, not a post-processing step

Say 'I don't know'

The single change that most improved trust was making the system decline to answer when retrieval confidence was low. Users forgive a system that admits a gap. They stop using one that has confidently misled them twice.

Citations are not decoration. Showing which document an answer came from lets a user verify in seconds, and it converts a black box into a research tool.

Evaluation has to be continuous

Knowledge bases drift. A retrieval configuration tuned in March degrades by August as documents are added and superseded. A held-out set of real questions with known-good sources, re-run on a schedule, is what tells you it is degrading before your users do.