Your RAG worked in the demo. Then real users arrived.

A few months back, my team picked up a project that sounded genuinely simple. Our support folks were drowning in repetitive tickets, the same questions every day, and every single answer already lived somewhere in our docs. Help articles, billing policies, a changelog stretching back years. All we needed was a thing that could read those docs and talk to users in plain language. No special magic, just connect the knowledge to the person asking.

We didn't want to fine-tune a model. Fine-tuning costs real money, and it goes stale the moment someone edits a help article, which around here was most days. It's also overkill when the answers are already written down. We didn't need the model to memorize our docs. We needed it to look things up at runtime.

That's RAG, retrieval-augmented generation. The idea is almost uncomfortably simple: chop your documents into chunks, turn each chunk into a vector (a long list of numbers that captures roughly what it means), and store them. When a question comes in, you convert it into a vector too, grab the chunks sitting closest to it, and paste those into the prompt as context. The model answers from what you handed it, not from whatever it half-remembers from training. No retraining. Edit a doc, re-embed it, done.

We had a first version running in an afternoon. In hindsight, that should have been my first warning.

The demo was perfect. Obviously.

We tested it the way you test something you built and are proud of. "What's our refund window?" Correct. "How do I reset my password?" Correct, steps in the right order. We even got clever, threw odd phrasings at it, tried billing edge cases, asked the same thing four different ways. It handled all of it. The demo for the wider team went great. Someone said the word "magic." We deployed that week.

Then real users showed up.

Within a few days, our internal Slack channel was a wall of screenshots. People posting bot answers with captions like "please tell me it didn't actually send this." The one that made me actually wince was a customer asking about refunds on a sale item, and the bot answering with total confidence, citing a fourteen-day return window that has never existed in any policy we've ever had. No hedging, no "I think." Just a number, delivered like fact.

So I stopped adding features and started tearing the thing apart.

Going in, I was sure the model was the weak link. It wasn't. It took me five separate problems to understand why, and the model turned out to be barely involved in any of them.

Issue 1: answers were always close, never right

The most common complaint wasn't wrong answers, it was useless ones. Someone would ask something specific and get back a paragraph that was sort of in the neighborhood but never actually landed on the fact. Close every time. Correct, never.

I started looking at what was getting retrieved, and the chunks were huge. We'd split the docs roughly by page, so each chunk was a whole section of mixed content. When someone asked about one specific sentence buried in that section, retrieval found the right page, but the chunk's vector was an average of everything on it. The one sentence that mattered got drowned out by the four hundred words around it.

I'd picked 512 tokens as the chunk size without thinking about it, because that's what every tutorial uses. So I cut it down to a few sentences each, added a little overlap at the boundaries so facts sitting at the edge of a chunk wouldn't get sliced in half, and tagged each chunk with its source article and section. Retrieval got sharper almost immediately.

The thing I took away from this: chunk size isn't a default you copy from a tutorial. It's a decision you make based on the questions your users are actually asking.

Issue 2: when it didn't know, it made something up

This one was meaner. Users would ask about things outside our docs entirely, and instead of saying so, the bot would confidently answer using whatever it happened to find.

The cause is obvious once you look: vector search always returns results. Ask for the top five chunks, you get five, every time, even when the best one is barely related. There's no built-in "none of these are good enough." So the model got five mediocre chunks and did what it's built to do, produced a helpful-sounding answer out of them. That's the thing about these systems: left alone, they would rather be confidently wrong than say nothing at all.

Two changes fixed this. First, a relevance floor: if the closest chunk doesn't clear a minimum similarity threshold, the bot says it doesn't have that information instead of guessing. That alone converted a pile of confident wrong answers into honest "I don't know"s. Our support team strongly preferred this, which tells you something.

Second, re-ranking. The first retrieval pass is fast but rough, good at pulling a broad set of candidates, bad at ordering them. So I widened the net to fifty results, then ran those through a slower model that scores each one against the actual question and reorders them. The best chunk started consistently landing where the model would use it.

One thing I got wrong here: I thought re-ranking would fix everything. It doesn't, because a re-ranker can only work with what you already retrieved. If the right chunk wasn't in the fifty, nothing brings it back. Missing answers are a retrieval problem. Wrong order is a ranking problem. I'd been treating them as the same issue.

Issue 3: compound questions always came back lopsided

Once the simple questions were solid, a new pattern showed up. Any question comparing two things, or asking about two at once, came back with a confident answer about one and barely a mention of the other. "How does Pro support compare to Enterprise?" got you three paragraphs on Pro and one vague sentence about Enterprise.

The cause: the question mentioned both plans, so the search looked for chunks similar to the whole query and landed somewhere in the middle, vaguely about both and specific to neither.

The fix was to stop treating compound questions as a single retrieval job. I added a step that detects when a question has multiple parts, runs a separate search for each one, and hands the model both result sets. One messy query became two clean ones. It costs an extra round-trip, and I'd pay that latency every time, because "compare X and Y" is exactly what someone asks when they're trying to make a decision. Being right there matters.

Issue 4: it was faithfully quoting policies we'd changed

The refund hallucination that kicked all of this off had two causes living together.

The first: we'd never told the model it could only answer from the retrieved context. The prompt just said "answer the question." So when the chunks didn't contain a sale-refund policy, the model did what it's trained to do and produced a plausible one. I rewrote the prompt to say it plainly: use only what's in the context, and if the answer isn't there, say so. That killed a big share of the made-up answers.

The second cause was more embarrassing. Some of the bad answers were from our docs. Just old versions. We'd indexed everything at launch, kept editing the live docs, and never re-indexed. The bot was quoting policies we'd changed months ago. So I set re-indexing on a schedule, wired it to trigger on doc edits, and started attaching timestamps so "current" actually meant current. Turns out a lot of what looked like hallucination was just the bot faithfully reading yesterday's newspaper.

Issue 5: I couldn't see what was happening

This is the one I should have built first, and ended up building last.

Every investigation above was harder than it needed to be because I had almost nothing to go on when something went wrong. A normal service throws a readable error. This thing handed me a confident paragraph and complete silence about how it got there.

So I added tracing. For every request, I logged the question, the chunks that came back, their scores, and the final answer. Now when a bad answer came in, I could open the trace and see where it broke. Was the right chunk retrieved and ranked poorly, or never retrieved at all? What used to cost an hour of guessing took about thirty seconds.

Then I added evals, the thing I'd been putting off because it felt like overkill for a "simple" project. I sat with our support team, collected forty real questions with known-correct answers, and built a test set. Before shipping any change, I could run all forty through the pipeline and get a score. The first time I ran it, I caught a "fix" that improved three answers and quietly broke five others. Without the eval set I'd have shipped it and heard about it from a customer.

After all of it

None of these fixes were clever. Smaller chunks. A relevance floor. Re-ranking. Query decomposition. A stricter prompt. Fresher data. Enough logging to actually see what the system was doing.

Notice what's not on that list: the model. We never touched it. Every fix was about what we put in front of it, not the thing doing the reading. I'd walked in expecting to tune the model and spent the whole time fixing retrieval instead. That gap, between the part everyone obsesses over and the part that actually breaks, turned out to be the entire lesson.

We redeployed with all of it. The confident nonsense mostly stopped. The "I don't know" rate went up, which sounds bad until you think about what the alternative was. An honest "I don't know" is worth more than a fluent lie. Our support team went from screenshotting failures to actually relying on it.

The demo had proved almost nothing. It showed the bot could answer questions we already knew the answers to, phrased the way we'd phrase them. Production was strangers asking things we'd never thought of, against docs that kept changing, and every failure lived in that gap. None of it was visible until real people showed up.

If I built another one tomorrow, I'd flip the order completely. Tracing and a small eval set before writing a single feature, so I could see what was happening and know whether changes were helping or hurting. The honest "I don't know" path before the always-has-an-answer path. It's not impressive to demo. It's the only reason the second version worked.

New to RAG, or want the fundamentals this story leans on? The plain-English explainer — What is RAG, really? — covers what RAG is, how it works, and when to use it. This piece is what production taught us beyond the basics.