---
title: 'RAG chunking strategies, tested on the same document set'
description: >-
  We ran eight RAG chunking strategies over our own blog with the same 24
  questions. Labelling each chunk with its title and heading beat making chunks
  bigger.
author: elroot_labs
date: '2026-11-17'
categories:
  - artificial-intelligence
---
The best RAG chunking strategy in our test was the plainest one: split on the document's own paragraphs and put the post title and section heading in front of each chunk before indexing it. That found the right section first for 14 of 24 questions while sending the model about 160 words per question. Large fixed-size windows found the right section somewhere in the top three more often, up to 22 of 24, but sent around seven times as much text, and their lead shrank once we added more documents. The rest of this post is the test, so you can run the same one on your own documents before choosing.

## What we tested, and on what

This test extends the retrieval experiment behind our post on [fine-tuning vs RAG](/blog/fine-tuning-vs-rag). The corpus is the 14 blog posts that were live on this site when that experiment ran, rebuilt from git history so the text is exactly what it was then. Rebuilding it reproduced the original baseline to the chunk: 451 paragraphs, with the right section ranked first for 10 of the 24 questions and in the top three for 13.

The 24 questions were written before the first run, each with the post and the section that answers it. Retrieval is BM25, a keyword-ranking method that needs no model, run with the standard settings. The only thing we changed between runs is how the text is split.

We counted a retrieved chunk as a hit when at least 30 of its words fall inside the target section, or all of them if the chunk or the section is shorter than that. This rule is generous to large chunks: a 400-word window that overlaps the right section by a paragraph counts the same as a paragraph that is exactly the answer. So next to every hit rate we report the words sent, the average size of the top three chunks, which is roughly what a RAG system would paste into the prompt for each question. A strategy that wins by sending the model a page and a half of text has not won much.

## Adding the title and heading did more than any chunk size

Here are the results on the 14 posts, 18,518 words in all. "Prefix" means the post title and section heading were added to each chunk's text before indexing.

| Strategy               | Prefix          | Chunks | Median words per chunk | Right section first | Right section in top 3 | Words sent per question |
| ---------------------- | --------------- | ------ | ---------------------- | ------------------- | ---------------------- | ----------------------- |
| Paragraph              | none            | 451    | 33                     | 10/24               | 13/24                  | 139                     |
| Paragraph              | title + heading | 451    | 33                     | 14/24               | 14/24                  | 160                     |
| 3-sentence windows     | none            | 405    | 43                     | 7/24                | 11/24                  | 146                     |
| 3-sentence windows     | title + heading | 405    | 43                     | 11/24               | 14/24                  | 151                     |
| Fixed 100 words        | none            | 192    | 100                    | 9/24                | 18/24                  | 287                     |
| Fixed 100 words        | title + heading | 192    | 100                    | 14/24               | 16/24                  | 287                     |
| Fixed 200 words        | none            | 100    | 200                    | 8/24                | 18/24                  | 564                     |
| Fixed 200 words        | title + heading | 100    | 200                    | 11/24               | 19/24                  | 561                     |
| Fixed 200, 50 overlap  | none            | 125    | 200                    | 8/24                | 14/24                  | 587                     |
| Fixed 200, 50 overlap  | title + heading | 125    | 200                    | 12/24               | 16/24                  | 589                     |
| Fixed 400 words        | none            | 54     | 400                    | 11/24               | 21/24                  | 1,063                   |
| Fixed 400 words        | title + heading | 54     | 400                    | 11/24               | 21/24                  | 1,073                   |
| Fixed 400, 100 overlap | none            | 64     | 400                    | 11/24               | 20/24                  | 1,105                   |
| Fixed 400, 100 overlap | title + heading | 64     | 400                    | 12/24               | 22/24                  | 1,114                   |
| Whole H2 section       | none            | 103    | 170                    | 10/24               | 14/24                  | 488                     |
| Whole H2 section       | title + heading | 103    | 170                    | 11/24               | 16/24                  | 522                     |

![Bar chart of RAG chunking strategies: how often the right section ranked first out of 24 questions, with and without a post title and heading prefix. Paragraph chunks with the prefix tied for best at 14 of 24, up from 10 without it; every strategy scored the same or better with the prefix.](/images/blog/elroot/figures/rag-chunking-strategies-right-section-ranked-first.png)

![Bar chart of words sent to the model per question for eight RAG chunking strategies. Paragraph chunks sent 139 to 160 words; fixed 400-word chunks with overlap sent over 1,100 words, seven to eight times as many.](/images/blog/elroot/figures/rag-chunking-strategies-words-sent-per-question.png)

The prefix moved paragraph chunks from 10 first-place hits to 14, for about 20 extra words per question. It helped every small-chunk strategy, and it helped least where the chunks were already large, which makes sense: a 400-word window usually contains its section's own vocabulary, and a 33-word paragraph often does not. A paragraph that says "run each one five times" never mentions evaluation or test cases. Its heading does.

This is the cheap, no-model version of what Anthropic calls [contextual retrieval](https://www.anthropic.com/news/contextual-retrieval). Their method has a model write 50 to 100 tokens of context for each chunk and prepends it before indexing, and they report it cut failed retrievals by 49% in their tests, or by 67% with a reranking step added. A title and a heading cost nothing to generate and are never wrong about which document a chunk came from. If your documents have usable headings, start there, and pay for generated context only if your own test shows the headings are not enough.

## Bigger chunks find more, and send far more text

Fixed 400-word windows put the right section in the top three for 21 of 24 questions, against 13 for plain paragraphs. That looks like the winner until you read the last column: 1,063 words per question against 139. Their first-place hit rate barely moved, from 10 to 11, so the gain comes from the generous hit rule and from each window covering more ground, while the ranking itself is no better at deciding which part of a post answers the question.

That trade has real costs. Every word sent is paid for on every question, and on an API priced per token the bill for this one design choice scales with the window. It also makes the model's job harder: the answer is now one sentence somewhere in three 400-word blocks, and the other two blocks may be confidently on a neighbouring topic. If the model is small or the answers must be exact, more context can produce worse answers.

There are cases where large chunks are right. If questions span several paragraphs, or the documents have no structure to split on, a window of a few hundred words may be the only way to keep an answer in one piece. Then measure the words sent as carefully as the hit rate.

## Overlap and sentence windows did not help here

Overlapping windows are a common default, on the theory that an answer cut in half at a boundary will survive whole in the next window. On this corpus they did not earn their keep. Fixed 200-word windows with a 50-word overlap scored 14 top-three hits against 18 without overlap, and at 400 words the overlap made a difference of one question either way. With 24 questions, gaps of one or two are within noise, so the honest reading is that overlap made no reliable difference while adding a quarter more chunks to store and search.

Three-sentence windows did worst of all, with 7 first-place hits. They cut across paragraph boundaries without regard to meaning, so a window often held the last sentence of one point and the first two of the next, and matched neither question well.

Whole sections split at each H2 heading looked promising, since our headings state the claim each section makes, and they were middling: 10 first-place hits, or 11 with the prefix. A section averages 170 words and often covers two or three related points, which dilutes the keyword match for any one of them.

## Retrieval got worse when we added eight more posts

A test set is only as good as its corpus, and corpora grow. So we added the eight posts published since the original test, taking the corpus to 22 posts and 28,772 words, and asked the same 24 questions. None of the new posts answers any of them; they are there as distractors.

Almost every strategy lost ground. Fixed 100-word windows fell from 18 top-three hits to 14, 400-word windows from 21 to 16, and whole sections from 14 to 12. Paragraphs with the title and heading prefix held at 14 in the top three, and lost one first-place hit, from 14 to 13. Large windows with the prefix held up best in the top three, 20 and 21, still at over 1,100 words per question.

The lesson is about when to test as much as what to choose. A chunking strategy tuned on today's documents is being tuned on the easiest version of the problem. Add the next six months of documents, or whatever stand-ins you have, before you trust the numbers.

## How to choose a chunking strategy for your own documents

This is the procedure we would follow on a client's documents, in this order:

1. Write 20 to 30 questions real users ask, and record the exact section that answers each one, before you run anything. Questions written after seeing results drift towards whatever the system already finds.
1. Start with chunks that follow the document's own structure: paragraphs, list items, table rows, sections. Research on [financial reports](https://arxiv.org/abs/2402.05131) found that chunking by structural element types improved RAG results over plain paragraph chunking, which matches what we saw with headings.
1. Prefix each chunk with its document title and heading path before indexing it.
1. Measure three numbers for each strategy: right section first, right section in the top three, and words sent. Judge them together.
1. Try one larger size to see whether your questions need it.
1. Add distractor documents and run it again.
1. Only after that, consider model-written chunk context, embeddings or a reranker, and test each one against the same question set so you know what it bought.

Our test has limits. It uses BM25 and not embeddings, 24 questions is a small set, and one company's blog is tidier than most document collections: consistent headings and short paragraphs favour structure-based chunks. The script, the questions and the raw output are in our evidence file, and the method transfers to any corpus in an afternoon. If your team is planning retrieval over its own documents and wants this run on them, [tell us what you are working with](/contact).
