How to chunk documents for RAG

Before a RAG system can retrieve anything, you have to decide what "anything" is. Retrieval does not return documents; it returns chunks. If the sentence that answers a question is split across two chunks, or buried in a 5,000-word chunk with ten other topics, retrieval struggles no matter how good your embedding model is.

The chunking trade-off

The goal is a chunk that answers one kind of question on its own.

Four common strategies

1. Fixed-size chunks

Cut every N characters or tokens, with an overlap.

def fixed_chunks(text, size=800, overlap=100):
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

Easy and predictable, but it cuts through sentences, code blocks and tables without caring. Useful as a baseline to beat.

2. Recursive splitting

Try to split on the largest natural boundary first, and only fall back to smaller ones when a piece is still too big: sections, then paragraphs, then sentences, then words. Many RAG libraries ship a version of this. It keeps paragraphs whole most of the time.

3. Structure-aware (heading-based) chunking

Use the document's own structure. For Markdown or HTML, split at h2 and h3 headings so each chunk is one section. If a section is too long, split it by paragraph inside the section.

This is what the tutor on this site does. Each chunk is one article section, and long sections are split at paragraph boundaries. Every chunk also carries its article title and heading path, for example:

What is a context window? > Budgeting a context window

That breadcrumb is added to the text that gets embedded, so a chunk that says "reserve room for the answer" still carries the fact that it is about context windows.

4. Semantic chunking

Embed each sentence, then start a new chunk wherever the meaning shifts sharply between neighbouring sentences. It can find topic boundaries in unstructured text such as transcripts, but it costs an embedding call per sentence and can produce very uneven chunk sizes.

Five habits that matter more than the strategy

1. Clean first. Navigation menus, cookie banners and repeated footers become junk chunks that match everything weakly.
2. Attach context to every chunk. Title, section heading, URL and date. Use the title and heading in the embedded text, and keep the rest as metadata for filtering and citations.
3. Keep atomic units whole. Do not split code blocks, tables or numbered procedures in the middle when you can avoid it.
4. Add modest overlap. Around 10 to 20 percent is a common start for fixed or recursive splitting.
5. Store a stable ID. Something like document-slug#section-heading, so you can re-index one document without rebuilding everything and so citations stay stable.

How to choose a size

Do not pick a number from a blog post, including this one. Measure:

1. Collect 30 to 50 real questions and note which section answers each.
2. Index your documents with two or three settings, for example 300, 600 and 1,000 tokens.
3. For each setting, check whether the right section appears in the top 3 or top 5 results. This is recall@k; see evaluating RAG systems.
4. Pick the smallest setting that keeps recall high.

You can explore the effect of size and overlap interactively in the chunking playground.

Summary

• Retrieval returns chunks, so chunking sets the ceiling on RAG quality.
• Prefer splitting on structure (headings, paragraphs) over blind fixed-size cuts.
• Attach titles and headings to every chunk, keep code and tables whole, and choose size by measuring recall on real questions.