What is chunking in RAG?
Chunking is the step in a retrieval-augmented generation pipeline that splits source documents into smaller passages before embedding and indexing, so search can return the passage that answers a query rather than a whole document.
Checked
Embedding models have input limits, and a vector for a 40-page contract says little about any one clause. So RAG pipelines cut documents into chunks, embed each one, and retrieve chunks rather than files.
The cut is easy to treat as a preprocessing detail. It is not. Chunk size, overlap and boundaries decide what a passage means on its own, and that decides whether the right passage can be found at all.
Common chunking strategies
Fixed token windows
Split every N tokens with some overlap. OpenAI's vector stores default to 800-token chunks with 400 tokens of overlap, adjustable from 100 to 4,096 tokens.
Structure-aware splitting
Split on headings, paragraphs and sentences first, falling back to smaller units only when a section is too long. Recursive character splitters work this way.
Semantic and LLM chunking
Group sentences by embedding similarity, or ask a model where the topic changes. More expensive to index, sometimes higher recall.
Late chunking
Embed the whole document with a long-context model first, then pool token vectors into chunks, so each chunk vector carries surrounding context. Described by Günther et al. at Jina AI in September 2024.
What the measurements show
Chroma's technical report Evaluating Chunking Strategies for Retrieval (2024-07-03) measured recall and precision at the token level and found strategies differed by up to 9% in recall. A recursive splitter at 200 tokens with no overlap was consistently strong, at about 88.1% recall. An LLM-based chunker reached the highest recall, about 91.9%. The 800-token, 400-overlap setting used by OpenAI's file search scored slightly below average on recall and lowest on the efficiency metrics.
The authors flag the limits: 472 generated queries over five corpora. The useful lesson is not a magic size but that chunking is a parameter worth measuring on real data.
Where chunking fails
Lost context
Anthropic's Contextual Retrieval post (2024-09-19) gives the classic case: a chunk saying revenue grew 3% that never names the company or the quarter. Its fix prepends 50 to 100 tokens of model-written context to each chunk, at a one-off cost Anthropic put at $1.02 per million document tokens with prompt caching.
Broken structure
Tables split across chunks, lists cut from their headings, and clauses that refer to "Section 4.2" lose their meaning in isolation.
Model input limits
Some retrieval models read only part of a long field. Elastic's ELSER, for example, encodes only the first 512 tokens, so longer text must be chunked below that size.
When to skip chunking
Anthropic's guidance is that a knowledge base under about 200,000 tokens can go straight into the prompt with prompt caching, no retrieval needed. Coding agents often skip chunked indexes too, reading files with search tools instead. Chunking earns its cost once the corpus is too large or too fast-growing to load whole.
How Motiontic's engineers apply it
Motiontic's engineers test two or three chunking strategies against the client's eval set rather than accepting a framework default, and look at the failures by hand: which answers were split, which tables were cut. Document structure usually decides the winner, so contracts, tickets and manuals rarely end up with the same settings.
Chunking strategies compared
| Fixed window | Structure-aware | Semantic or LLM | Late chunking | |
|---|---|---|---|---|
| Splits on | Token count | Headings and paragraphs | Topic shifts | Token count, after embedding |
| Index-time cost | Lowest | Low | Higher | Needs a long-context embedder |
| Keeps context | Only via overlap | Within a section | Within a topic | From the whole document |
| Typical failure | Cuts mid-sentence or mid-table | Uneven chunk sizes | Slow and harder to debug | Limited by the model's context |
Common questions
What is a good chunk size for RAG?
There is no universal value. Chroma's 2024 study found a 200-token recursive split with no overlap consistently strong on its data, but the right size depends on the documents and should be measured on real queries.
Does chunk overlap help retrieval?
It can stop answers from being cut in half, but it duplicates tokens. In Chroma's study, reducing overlap improved token efficiency, and the high-overlap default scored lowest on those measures.
What is the difference between chunking and contextual retrieval?
Chunking decides where documents are cut. Contextual retrieval adds a short, model-written description of the surrounding document to each chunk before indexing, so a cut chunk keeps its meaning.
Is chunking needed with long context windows?
Not always. Anthropic suggests a knowledge base under about 200,000 tokens can be placed in the prompt directly. Larger or fast-changing corpora still need chunking and retrieval.
Related
Contextual retrieval
Give every chunk a line of context before it is indexed, so it can still be found.
Embeddings
Meaning turned into vectors, so similar things can be found by distance.
Retrieval-augmented generation (RAG)
Fetch the right passages at query time, then answer from them with citations.
Multimodal RAG
RAG over scans, slides, charts, audio and video, not only extracted text.
AI product development
New products and major AI features, built from scratch alongside the client's team.
Sources
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.