Choosing chunk size and overlap for RAG

Last updated 2026-10-14 · by Prajyot Mulik

Try the tool: RAG Text Chunker

Retrieval-augmented generation (RAG) splits your documents into chunks, stores an embedding for each chunk, and at question time retrieves the best few chunks to put in the model’s prompt. Two settings, chunk size and overlap, quietly shape the whole system. There is no universally correct value, and any guide that gives you one number is guessing. What this guide can do is show you what each setting trades off, give you the arithmetic to predict cost and context use, and describe a way to choose by measurement. You can try every setting in the RAG Text Chunker.

What chunk size controls

A chunk is the unit that gets embedded, retrieved and pasted into the prompt. Its size therefore affects three things at once.

Match precision. An embedding summarises everything in the chunk. A small chunk about one idea produces a focused embedding that matches a focused question well. A large chunk covering several ideas produces a blurred embedding, so a specific question may match it weakly even though the answer is inside.

Context for the answer. A small chunk may cut off the sentence that gives the answer meaning: a definition without its caveat, a table row without its header. A larger chunk keeps surrounding context together, which helps the model answer correctly.

Prompt budget and cost. Retrieving k chunks of size s puts roughly k × s tokens into every prompt. Larger chunks fill the context window faster and raise the per-call cost, which you can estimate with the token cost calculator. Smaller chunks let you retrieve more of them for the same budget.

So the choice is rarely “small is better” or “large is better”. It is a balance between precision and context that depends on your documents and questions.

What overlap does

Overlap repeats the end of one chunk at the start of the next. Its purpose is to protect against a boundary falling in the middle of an important passage: if a fact straddles the boundary, the overlap makes sure it appears whole in at least one chunk.

Overlap is not free. Every overlapped character is embedded, stored and possibly retrieved twice. With a chunk size of 1,000 and an overlap of 200, each new chunk advances only 800 characters, so you store 25% more chunks than you would with no overlap. Two overlapping neighbours can also both be retrieved for the same question, spending prompt budget on near-duplicate text. A small overlap, often a fraction of the chunk size, is the usual compromise; whether it helps you is something to test rather than assume.

The arithmetic

For a text of L units split into chunks of size s with overlap o, each chunk starts s − o units after the previous one, and the number of chunks is ⌈(L − s) ÷ (s − o)⌉ + 1 for L larger than s. Examples for a 1,000,000-character corpus:

Halving the chunk size roughly doubles the chunk count, and with it the number of vectors to store and search. The vector database size calculator turns a chunk count into RAM and disk.

Characters, tokens or sentences?

The chunker offers three units, each with a different failure mode.

• Characters are simple and predictable, but a chunk can end mid-word or mid-sentence. Embedding models and prompts are limited in tokens, not characters, so character sizes only approximate the token budget (English is often around four characters per token, but code and other languages differ).
• Tokens make the size match what models actually count, so a chunk size of 400 tokens really is 400 tokens for the tokenizer in use. The tool uses the o200k_base encoding; if your embedding model uses a different tokenizer, counts will differ. A token window can also start or end inside a multi-byte character.
• Sentences keep sentence boundaries intact, which usually reads better and avoids cutting a sentence in half. Sizes vary with sentence length, and sentence detection is heuristic: abbreviations and text without punctuation can split unexpectedly.

None of these understands headings, lists, code blocks or tables. If your documents have strong structure, splitting along that structure (sections, then paragraphs) and using the size limit only as a safety net often produces more meaningful chunks than any fixed window. That requires structure-aware tooling this page does not provide.

A method that works: measure it

1. Write test questions. Collect 30 to 100 real questions your users will ask, and for each note which passage in your documents answers it.
2. Try a few settings. For example three sizes and two overlaps. Chunk with the same tool each time so results are comparable.
3. Measure retrieval, not generation. For each question, check whether the right passage appears among the top k retrieved chunks. This hit rate is cheap to compute and isolates the retrieval step.
4. Then check answers. Take the best one or two settings and read the generated answers for a sample of questions. Look for answers cut short by missing context, and for answers confused by too much irrelevant text.
5. Include cost. Compare the chunk counts, the tokens added to each prompt, and the resulting monthly cost, not just quality.
6. Re-test when things change. A different embedding model, document type or question style can move the best setting.

Practical starting points (not rules)

If you have no evidence yet, start with moderate chunks, a modest overlap, and retrieve a handful of chunks, then let the measurements above move you. Keep a note of the settings and the test set so you can repeat the comparison later. The share link on the chunker saves your settings in the address so you can send exactly what you tried to a colleague.

Sources

• tiktoken – the o200k_base tokenizer used by the chunker’s token mode (read on 14 October 2026)
• MDN: Intl.Segmenter – the sentence segmentation used by the chunker’s sentence mode (read on 14 October 2026)
• The chunk-count formula and the table above are simple arithmetic; the same logic is covered by this site’s automated tests.