Fixed-size, recursive, semantic, and contextual chunking split a document for retrieval in four genuinely different ways, and the choice changes how often your RAG system retrieves the wrong passage. Fixed-size chunking is fastest to build but breaks context at arbitrary boundaries; recursive chunking respects the document’s own structure; semantic chunking splits wherever the meaning actually shifts; contextual chunking adds a short explanatory note to each piece before indexing it. Anthropic has published real measurements of how much that last approach reduces retrieval failures, which makes this one of the few areas of RAG design where the trade-offs aren’t just opinion.
Fixed-size chunking: split by token or character count
Fixed-size chunking cuts a document into pieces of a set length — commonly a few hundred tokens, with a small overlap between consecutive chunks so a sentence split across a boundary still appears in full somewhere. Anthropic’s own description of a typical RAG pipeline puts chunk size at “usually no more than a few hundred tokens” before each piece is embedded and indexed.
The appeal is implementation simplicity: a fixed-size splitter needs no understanding of the document’s structure, so it runs identically on a PDF, a markdown file, or a plain-text export. The cost is that it has no idea where a sentence, a table row, or a section actually ends. A chunk boundary can land in the middle of a clause, and the resulting chunk — taken on its own, which is how it will be retrieved — can lose the information needed to make sense of it.
Recursive chunking: split along the document’s own structure first
Recursive chunking tries a list of separators in order of priority — typically double newlines (paragraph breaks), then single newlines, then sentence boundaries, then, only as a last resort, a hard character count. It splits on the first separator that produces chunks under the target size, and only falls back to a cruder split where the structure itself doesn’t cooperate (a single paragraph longer than the limit, for instance).
This keeps most chunks aligned with something a human would recognize as a complete thought — a full paragraph, or a full list item — rather than an arbitrary slice. It doesn’t solve the context-loss problem fixed-size chunking has, though: a self-contained paragraph can still fail to mention which company, product, or time period it’s describing, because that information lived in a heading or an earlier paragraph the chunk no longer includes.
Semantic chunking: split where the meaning shifts, not at a fixed point
Semantic chunking drops the idea of a target chunk size entirely and instead splits based on where the content’s meaning actually changes. LlamaIndex’s documentation for its SemanticSplitterNodeParser describes the mechanism directly: rather than splitting at a fixed size, it “adaptively picks the breakpoint in-between sentences using embedding similarity,” so that each resulting chunk groups sentences that are semantically related to each other and separates from sentences that aren’t.
In practice this means consecutive sentences are embedded and compared, and a split is inserted wherever the similarity between neighboring sentences drops enough to suggest a topic change — configured in LlamaIndex’s implementation through a percentile threshold rather than a hard distance cutoff. The documentation is explicit about two real limitations: the sentence-boundary detection is tuned mainly for English text, and the similarity threshold used to decide where a “real” topic break occurs typically needs tuning per corpus rather than working well at one default setting across every document type.
That makes semantic chunking the approach most likely to produce chunks that are topically coherent on their own merits — useful when a document covers several unrelated subjects back to back with no clean heading structure to split on — at the cost of an embedding call per sentence at indexing time and a threshold that needs validating against your own documents rather than trusted as a sensible default.
Contextual chunking: add the missing context back in before indexing
Contextual chunking, the technique behind Anthropic’s Contextual Retrieval method, treats the context-loss problem directly instead of trying to avoid it through smarter splitting. Before each chunk is embedded, a short, chunk-specific explanatory note is generated and prepended to it — identifying, for example, which company and which fiscal quarter a chunk of a financial filing refers to, information that may have appeared only once, in a heading several paragraphs earlier.
Anthropic’s published example makes the failure mode concrete: a chunk that reads only “The company’s revenue grew by 3% over the previous quarter” doesn’t say which company or which quarter, which makes it hard to retrieve when a query asks about a specific company, and hard to use correctly even if it is retrieved. Prepending a sentence of context — naming the company and the quarter — fixes both problems without changing what information the chunk contains.
Before and after, the difference looks like this:
Before (fixed-size or recursive chunk, taken in isolation): "The company's revenue grew by 3% over the previous quarter." After (contextual chunk, same underlying sentence): "This chunk is from an SEC filing on ACME Corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."
Anthropic measured the effect of this at scale, using the top-20-chunk retrieval failure rate (how often the correct chunk isn’t among the top 20 results returned) as the benchmark, starting from a 5.7% baseline failure rate: Contextual Embeddings alone reduced that failure rate by 35% (to 3.7%), and combining Contextual Embeddings with Contextual BM25 keyword search reduced it by 49% (to 2.9%). Those are the only numbers in this comparison that come from a controlled measurement rather than general reasoning about how each method behaves, which is why they’re worth citing directly rather than paraphrasing into a vaguer claim.
These four aren’t strictly exclusive choices. Contextual chunking is an addition made on top of a base splitting method, not a replacement for one: a pipeline still has to decide how to produce the initial chunks — by fixed size, by recursive structure, or semantically — before generating and prepending the explanatory context to each one. In practice, a contextual pipeline is usually recursive or semantic chunking plus a context-generation step, not a fourth, unrelated splitting algorithm. Anthropic’s write-up notes that prompt caching is what keeps that added generation step affordable when it’s run across an entire corpus, since each chunk’s surrounding document context can be cached rather than re-processed from scratch for every single chunk.
The four approaches, side by side
| Approach | How it splits | Main risk | Extra cost |
|---|---|---|---|
| Fixed-size | Hard token or character count, with overlap | Boundaries fall mid-sentence or mid-table; isolated chunks can lose identifying context | None beyond the base embedding cost |
| Recursive | Tries paragraph, then line, then sentence separators before falling back to a character count | Chunks respect structure but still lose context that lived outside the chunk (a heading, an earlier paragraph) | Negligible — splitting logic only, no extra model calls |
| Semantic | Embeds consecutive sentences and splits where similarity drops past a tuned percentile threshold | Threshold needs per-corpus tuning; sentence-boundary detection is tuned mainly for English | One embedding call per sentence at indexing time |
| Contextual | Recursive or fixed-size split, then a short explanatory note generated per chunk and prepended before embedding | Adds a generation step per chunk, so re-chunking a large corpus has a real, ongoing cost | One model call per chunk at indexing time (Anthropic’s write-up notes prompt caching keeps this affordable at scale) |
Picking one, and the mistake that undoes any of them
For a small, stable knowledge base, recursive chunking with a reasonable overlap is usually enough: the structure-aware splitting avoids the worst boundary problems, and the corpus is small enough that a human can spot-check chunks for missing context. Contextual chunking earns its extra cost on larger or more homogeneous corpora — financial filings, legal contracts, long technical manuals — where many chunks share the same surface-level language and differ only in which entity or time period they describe, which is exactly the condition under which a chunk read in isolation becomes ambiguous.
Whichever method is chosen, one mistake undoes the benefit of any of them: re-indexing a chunking pipeline only when it’s first built, and never again. Source documents get edited, headings get renumbered, and a chunking boundary that made sense against last quarter’s filing can land somewhere nonsensical against this quarter’s version. Treat the chunk size, the separator list, and — if using contextual chunking — the context-generation prompt as configuration that needs to be re-validated whenever the shape of the source documents changes materially, not as a one-time setup step.
Chunking only matters in service of the retrieval step built on top of it. For the fundamentals of how those retrieved chunks get stored and searched, see the guide to vector databases and the explanation of what embeddings actually are. The plain-English guide to retrieval-augmented generation covers how chunking fits into the full RAG pipeline, and the guide to context windows is the related constraint on the other end of the pipeline: how much of what you retrieve the model can actually use at once.
Featured image: “Wikimedia Foundation Servers 2015-88” by Victor Grigas, CC BY-SA 3.0.
