How RAG works
RAG has two steps: retrieve relevant passages from your data for the question, then have the model generate an answer using only those passages. Good retrieval yields 95–99% domain accuracy; bad retrieval makes even a great model wrong.
Retrieval is the hard part
Modern models generate well; the failure point is fetching the right context. If retrieval returns irrelevant chunks, the model answers faithfully from bad information, which is why retrieval, not generation, is where teams should invest.
Retrieval quality depends on how you chunk documents, how you embed and index them, and how you rank results, unglamorous engineering that determines everything downstream. Yet roughly 70% of RAG systems have no systematic evaluation of whether retrieval is working, so teams tune the model when the real problem is the index. Measuring retrieval precision first is the fastest path to a trustworthy system.
Frequently asked questions.
What makes RAG fail in production?
Almost always retrieval: poor chunking, weak indexing, or no ranking, so the wrong passages reach the model. Fixing retrieval quality resolves most production RAG failures faster than changing the model.