85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is chunking in RAG?

Chunking is the step in a retrieval-augmented generation pipeline that splits source documents into smaller passages before embedding and indexing, so search can return the passage that answers a query rather than a whole document.

Checked

Embedding models have input limits, and a vector for a 40-page contract says little about any one clause. So RAG pipelines cut documents into chunks, embed each one, and retrieve chunks rather than files.

The cut is easy to treat as a preprocessing detail. It is not. Chunk size, overlap and boundaries decide what a passage means on its own, and that decides whether the right passage can be found at all.

Common chunking strategies

  • Fixed token windows

    Split every N tokens with some overlap. OpenAI's vector stores default to 800-token chunks with 400 tokens of overlap, adjustable from 100 to 4,096 tokens.

  • Structure-aware splitting

    Split on headings, paragraphs and sentences first, falling back to smaller units only when a section is too long. Recursive character splitters work this way.

  • Semantic and LLM chunking

    Group sentences by embedding similarity, or ask a model where the topic changes. More expensive to index, sometimes higher recall.

  • Late chunking

    Embed the whole document with a long-context model first, then pool token vectors into chunks, so each chunk vector carries surrounding context. Described by Günther et al. at Jina AI in September 2024.

What the measurements show

Chroma's technical report Evaluating Chunking Strategies for Retrieval (2024-07-03) measured recall and precision at the token level and found strategies differed by up to 9% in recall. A recursive splitter at 200 tokens with no overlap was consistently strong, at about 88.1% recall. An LLM-based chunker reached the highest recall, about 91.9%. The 800-token, 400-overlap setting used by OpenAI's file search scored slightly below average on recall and lowest on the efficiency metrics.

The authors flag the limits: 472 generated queries over five corpora. The useful lesson is not a magic size but that chunking is a parameter worth measuring on real data.

Where chunking fails

  • Lost context

    Anthropic's Contextual Retrieval post (2024-09-19) gives the classic case: a chunk saying revenue grew 3% that never names the company or the quarter. Its fix prepends 50 to 100 tokens of model-written context to each chunk, at a one-off cost Anthropic put at $1.02 per million document tokens with prompt caching.

  • Broken structure

    Tables split across chunks, lists cut from their headings, and clauses that refer to "Section 4.2" lose their meaning in isolation.

  • Model input limits

    Some retrieval models read only part of a long field. Elastic's ELSER, for example, encodes only the first 512 tokens, so longer text must be chunked below that size.

When to skip chunking

Anthropic's guidance is that a knowledge base under about 200,000 tokens can go straight into the prompt with prompt caching, no retrieval needed. Coding agents often skip chunked indexes too, reading files with search tools instead. Chunking earns its cost once the corpus is too large or too fast-growing to load whole.

How Motiontic's engineers apply it

Motiontic's engineers test two or three chunking strategies against the client's eval set rather than accepting a framework default, and look at the failures by hand: which answers were split, which tables were cut. Document structure usually decides the winner, so contracts, tickets and manuals rarely end up with the same settings.

Chunking strategies compared

Chunking strategies compared
Fixed windowStructure-awareSemantic or LLMLate chunking
Splits onToken countHeadings and paragraphsTopic shiftsToken count, after embedding
Index-time costLowestLowHigherNeeds a long-context embedder
Keeps contextOnly via overlapWithin a sectionWithin a topicFrom the whole document
Typical failureCuts mid-sentence or mid-tableUneven chunk sizesSlow and harder to debugLimited by the model's context

Common questions

What is a good chunk size for RAG?

There is no universal value. Chroma's 2024 study found a 200-token recursive split with no overlap consistently strong on its data, but the right size depends on the documents and should be measured on real queries.

Does chunk overlap help retrieval?

It can stop answers from being cut in half, but it duplicates tokens. In Chroma's study, reducing overlap improved token efficiency, and the high-overlap default scored lowest on those measures.

What is the difference between chunking and contextual retrieval?

Chunking decides where documents are cut. Contextual retrieval adds a short, model-written description of the surrounding document to each chunk before indexing, so a cut chunk keeps its meaning.

Is chunking needed with long context windows?

Not always. Anthropic suggests a knowledge base under about 200,000 tokens can be placed in the prompt directly. Larger or fast-changing corpora still need chunking and retrieval.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.