Your retrieval system finds the right document. It even finds the right page. And the answer it gives back is still somehow off — close to the fact, sitting right next to it, but never quite landing on it. You ask about one specific clause and get a paragraph that's vaguely about the whole section it lived in.
Most people, at this point, go looking for a better embedding model or a smarter prompt. The actual culprit is usually further upstream, in a step almost nobody thinks about twice: how the documents got cut into pieces in the first place.
If you take one idea from this page, take this: you don't retrieve documents. You retrieve chunks. When a question comes in, your system can only hand the model a whole chunk, never half of one and never two stitched together. So the chunk isn't a storage detail. It's the smallest unit your model ever sees, and where you draw its boundaries sets the resolution of the whole system.
Which means chunking really comes down to one question: where do you put the scissors?
What a chunk is
A chunk is a slice of a document that gets turned into a single embedding: one vector, one point in space, one searchable thing. Cut a help article into eight chunks and you've made eight separately findable units. The model never searches the article; it searches those eight pieces and gets back whichever ones sit closest to the question.
That's the whole reason the cut matters so much. A chunk's embedding represents the average meaning of everything inside it. One tight chunk about refund windows points cleanly at refund questions. A chunk that crams the refund policy, the shipping schedule, and a support phone number into one piece points vaguely at all three and sharply at none.
So a good chunk has a simple property: it holds one complete thought, and it can stand on its own. Small enough to be about one thing. Big enough that the one thing still makes sense when you read it cold, with no surrounding paragraphs to lean on.
How chunking works
There are only a few knobs, and they all serve that one property.
Size. How much text goes in each chunk, usually measured in tokens. This is the knob everyone reaches for first and understands least. There's no universal right answer, and the popular default (somewhere around 500 tokens, because that's what the first tutorial you read used) is just that: a default, not a decision.
Overlap. Let each chunk share a little text with the one before it, so a fact sitting right on a boundary doesn't get sliced clean in half. Without overlap, a sentence that straddles the cut can end up half in one chunk and half in another, and fully useful in neither.
Boundaries. Where you actually make the cut. The lazy version is fixed-size: count 500 tokens, cut, repeat, regardless of what you slice through. Better is to cut on the document's own seams — paragraph breaks, sentence ends, markdown headings. If your documents have structure, that structure is a gift; it's the author already telling you where one idea stops and the next begins.
Metadata. Tag each chunk with where it came from: source document, section heading, date. A chunk torn loose from its origin can still answer a question, but it can't tell you which policy it's quoting or when that policy was true. Attach the labels at cut time, because you can't recover them later.
Where it breaks
Almost every chunking failure is one of these, and you can usually diagnose which from the symptom.
• Chunks too big. The meaning averages out. The one sentence that answers the question gets drowned by the four hundred words around it, so retrieval finds the right region and still misses the fact. This is the "close but never right" symptom, and it's the single most common one. We watched it play out on a live support bot, in detail, in the war story — chunk size was the first thing that had to change.
• Chunks too small. Now you've got the opposite problem. A lone sentence retrieves cleanly but arrives stripped of context: "It must be returned within that window" is useless when "it" and "that window" lived in the sentence you cut away.
• Cutting through the middle of an idea. Split an answer across two chunks and retrieval can only ever fetch one of them. The complete answer existed in your documents; your scissors just made it unfindable.
• No overlap. Facts that live at chunk boundaries get sliced and lost. Cheap to prevent, annoying to debug.
• One size for everything. A dense API reference and a rambling blog post do not want the same chunk size, and forcing one number across both guarantees you're wrong for at least one of them.
• No metadata. The bot gives a confident answer and can't cite a source, or quotes a policy without knowing it's two versions out of date. Often what looks like hallucination is just a chunk that lost its label.
The pattern underneath all of these: a re-ranker can't reorder a chunk that was never retrieved, and an embedding model can't represent an idea that got split in half. Nothing downstream can recover an answer you already sliced apart. That's why this step, boring as it looks, sets the ceiling for everything after it.
Choosing how to cut
You don't guess your way to a chunking strategy. You match it to two things: the shape of your content, and the shape of your questions.
Start with the content. Dense reference material, where one line is one fact, wants small, tight chunks. Narrative or explanatory writing, where meaning builds across a few sentences, wants larger ones that keep a thought intact. When the document has real structure (headings, sections, list items), cut on that structure first and worry about token counts second.
Then look at your questions. If people ask narrow, specific things ("what's the refund window on sale items"), you need chunks fine enough to isolate that fact. If they ask broad, synthesizing things ("how does your returns process work"), chunks that hold a whole section serve better.
When those two pull in opposite directions, there's a technique worth knowing, sometimes called small-to-big or parent-document retrieval: embed small chunks so matching is precise, but when one hits, hand the model the larger passage around it. You get accurate retrieval and full context at the same time, instead of trading one for the other. It's the closest thing chunking has to a free lunch.
And whatever you pick, measure it. Build a set of real questions with known answers, change the chunking, and check whether retrieval got better or worse. Chunk size is exactly the kind of change that feels right and quietly makes things worse, and the only way to know is to test it. (More on that when we reach evals.)
Where to go next
Chunking is the first real decision inside the knowledge retrieval mental model. The pieces that follow build directly on it:
• Embeddings — what each chunk actually becomes, and why similar meanings land near each other
• Vector databases — where those embeddings live and how the search works
• Re-ranking — getting the best retrieved chunk in front of the model, not just a relevant one
• Evals — proving a chunking change helped instead of hoping it did
• then the project: build a production RAG, end to end.
If you want the fundamentals this leans on, start at What is RAG, really?. And if you want to watch a chunking mistake cause real damage and then get fixed, that's the war story: Your RAG worked in the demo. Then real users arrived.
Chunking looks like a preprocessing step you set once and forget. It's really the resolution dial for the whole system. Set it well and every step after it has a chance. Set it badly and no embedding model, no re-ranker, and no amount of prompt-tuning will bring back the answer you already cut in half.
