You've got a language model that can write a passable sonnet about almost anything, and you ask it one simple question about your own company: "what's our refund window?" but it has no idea about your company to answer. As it was trained on a slice of the public internet up to some cutoff date, your refund policy never made it up there. Neither was last week's changelog, your internal runbook, or the contract you signed yesterday.
That's the gap RAG closes. The model is brilliant at reasoning over text and useless at knowing your specific facts. So you stop asking it to know them, and hand them over at the moment you ask the question.
The whole idea in one line: RAG turns a closed-book exam into an open-book one. Instead of hoping the model memorized the answer, you find the relevant pages and put them in front of it, then ask it to answer from those. "Retrieval" is finding the right pages; "augmented generation" is the model writing an answer using them. Everything else you'll read about, every acronym and tool, is plumbing in service of finding the right pages and handing them over cleanly.
If you keep one picture from this page, make it this: a RAG system is a librarian, not a know-it-all. It doesn't hold all knowledge in its head. It goes and finds the right passage, then reads it back to you in plain language.
And here's the part that surprises people the first time they ship one: the language model is rarely where RAG goes wrong. The model can already read and reason perfectly well. The hard part is finding the right passage to hand it — and that's where almost every real-world failure actually lives. Hold onto that. It quietly explains most of what follows.
Why not just teach the model? (RAG vs fine-tuning)
The obvious alternative is to train the model on your documents, which is called fine-tuning. For knowledge, it's usually the wrong tool, for two reasons.
It's frozen in time. The moment someone edits a help article, a fine-tuned model is out of date, and the only fix is to train it again. RAG just re-reads the current document.
And fine-tuning teaches behavior, not facts. It's great for "always answer in this format" or "adopt this tone." It's bad and expensive for "know these ten thousand constantly-changing facts." So the rule of thumb: fine-tune to change how the model behaves; use RAG to change what it knows. They aren't rivals; mature systems often do both.
How it actually works
A RAG system has two phases, and keeping them separate in your head clears up most of the confusion.
Phase one happens ahead of time (ingestion). You cut your documents into smaller pieces called chunks. Each chunk becomes an embedding: a list of numbers, a vector, that represents the chunk's meaning as a point in space, with the useful property that chunks meaning similar things land near each other. You store these vectors in a vector database so you can search them fast.
Phase two happens when a question arrives (retrieval and generation). You embed the question the same way, ask the database for the chunks whose vectors sit closest to it, and paste those "relevant pages" into the prompt with an instruction like "answer using only the information below." The model reads the context and writes the answer, ideally pointing at which chunk it used.
Stripped down, every RAG system is that loop: embed the question, find the nearest chunks, hand them to the model, get a grounded answer. Chunking strategies, re-rankers, hybrid search, vector databases, they're all just ways to make one of those steps less wrong.
![[ DIAGRAM → the RAG loop ] — Question → Embed → Vector DB (find nearest chunks) → Retrieved context → LLM → Answer. Replace this line with the diagram image when pasting.](https://res.cloudinary.com/dmcl2fqye/image/upload/v1782655838/til-insights/img-1782655838415-981fc06e-ea08-47ee-b9c4-e98e63d112cc.png)
Where it breaks (and why)
RAG is easy to stand up and surprisingly hard to make reliable, because the weak link isn't the model, it's the retrieval. Fetch the wrong pages and the model will write a confident, fluent answer from the wrong pages, and it'll sound exactly as sure as a right one.
The usual failure points, in plain terms:
• Bad chunking. Pieces too big and the meaning averages into mush; too small and you lose the context that made a sentence mean anything.
• "Retrieved" isn't "relevant." Vector search always hands back something, even for a question your docs can't answer. With no relevance cutoff, the model fills the gap with a plausible invention.
• Stale data. Index once, let the docs change, and the system will faithfully quote a policy nobody follows anymore.
• Multi-part questions. "Compare A and B" needs facts from two places; one search aimed at the whole question grabs a blurry middle that's about both and good for neither.
None of these are exotic. They're the default behavior, invisible until real users arrive. We took a production system through every one of them, screenshots and all, in the companion piece: Your RAG worked in the demo. Then real users arrived. That's the war story; this page is the map.
When to reach for RAG (and when not to)
Reach for RAG when the answers live in your data the model never saw, that data changes, you need answers grounded in sources a user can check, or the corpus is too big to paste into the prompt.
Skip it when you mainly need the model to behave differently (that's fine-tuning), when the relevant information is tiny and static (just put it in the prompt and skip the machinery), or when the question doesn't need outside facts at all.
Where to go next
This page is the entrance to one mental model: knowledge retrieval, how a system finds the right context to answer from. Each piece below sharpens one part of the loop:
• Chunking — how to cut documents so the right thing stays findable
• Embeddings — what a vector actually represents
• Vector databases — where the vectors live and how search works
• Re-ranking — getting the best chunk in front of the model, not just a relevant one
• Evals — knowing whether any of it actually works
• then the project: build a production RAG, end to end.
Once you see RAG as finding the right pages before answering, the rest stops looking like magic. Chunking, embeddings, vector search, re-ranking: none of it is mysterious. Each one is just another way to help the librarian find the right shelf faster.
And if you want to watch every one of these problems show up in a real system, and then get fixed one at a time, that's the war story: Your RAG worked in the demo. Then real users arrived. This page is the map. That one is the territory.
