Chunking Strategies for RAG: Size, Overlap, Structure & Semantic Splitting

Imagine tearing a 300-page manual into pieces and putting each piece in a separate envelope. Later, someone asks a question and you must hand them three envelopes — no more. How you tore the pages now decides whether they get an answer or confetti. That's chunking. It looks like a boring preprocessing detail; it is quietly the setting that decides how good your RAG system will ever be, and interviewers ask about it precisely because it exposes whether you've built one.

Why not just embed each document whole?

Two reasons, both fatal. One — the vector goes blurry. An embedding is a single point of meaning. A 50-page HR manual covers leave, payroll, conduct, IT policy and travel; one vector for all of it is the average of five topics, which sits near none of them. Ask about leave and it may lose to a small, focused chunk from a totally different document.

Two — you must paste it into the prompt. Retrieval returns text that goes into a limited, per-token-billed context window. Handing the model 50 pages to answer one question is the "paste everything" failure from lesson 1, just wearing a RAG costume.

So: chunks — pieces small enough to be focused, big enough to make sense alone.

The size trade-off (there is no universal number)

A sane starting point for prose is 200–500 tokens with 10–20% overlap — then tune it by measuring recall@k on a golden set, never by vibes (lesson 9). Chunk size is a hyperparameter of your corpus, not a constant of the universe: dense legal clauses and chatty support transcripts want different numbers.

Overlap — insurance against bad luck

Look at the top row. The sentence "the notice period for confirmed employees is 30 days" straddles a boundary: chunk 2 ends with "the notice period for" and chunk 3 begins with "confirmed employees is 30 days". Neither chunk matches a question about notice periods well, so the fact is effectively invisible to your bot — and nobody will ever know why.

Overlap fixes it: let each chunk repeat the last ~10–20% of the previous one, so boundary facts survive whole in at least one chunk. You pay with a slightly larger index and some duplicate text in retrieval. Worth it every time.

Wait — why am I choosing boundaries at all? The document has them

Here's the shift that upgrades a pipeline. Most real documents already contain human-designed boundaries: headings, numbered clauses, list items, API endpoints, Q&A pairs. Splitting by character count through a document that has headings is throwing away free information.

Structure-aware chunking splits on those boundaries first, and falls back to fixed-size + overlap only inside sections that are too long. One clause = one chunk. One endpoint = one chunk. And crucially, keep the heading attached to its text, so "12 days per year" arrives with "§4 Casual Leave" rather than floating free.

import re

def split_by_headings(markdown_text, max_tokens=400):
    """Split on markdown headings first; only then by size."""
    sections = re.split(r"\n(?=#{1,3} )", markdown_text)
    chunks = []
    for section in sections:
        heading = section.split("\n", 1)[0].strip("# ")
        if approx_tokens(section) <= max_tokens:
            chunks.append(section)
        else:
            for part in chunk_words(section, size=max_tokens, overlap=50):
                chunks.append(f"{heading}\n{part}")   # keep the heading!
    return chunks

§4 Casual Leave
Employees are entitled to 12 casual leaves per calendar year…
(heading travels with every piece of an over-long section)

Semantic chunking — split where meaning shifts

What about documents with no headings — meeting notes, transcripts, scraped pages? Semantic chunking finds the boundaries itself: embed consecutive sentences, and cut where the similarity between neighbours drops sharply, because that dip is a topic change.

It produces noticeably cleaner chunks on unstructured prose. The honest cost: it embeds everything an extra time during indexing, and it's a one-off expense you pay per re-index. For structured documents it's usually unnecessary — the headings were already better boundaries than any model will infer.

Enriching chunks — give the fragment its bearings

A chunk that reads "this must be approved by the reporting manager" is nearly useless: approved by whom, for what? The fix is to prefix each chunk with a little context before embedding:

enriched = (
    f"Document: {doc_title} | Section: {section_path}\n"
    f"Effective: {effective_date}\n\n"
    f"{chunk_text}"
)
# embed(enriched) - and store the same text for the prompt

Two benefits at once: the embedding now carries the document's topic (so it's more findable), and the LLM sees where the text came from (so it can cite it and won't confuse two similar policies). The cost is extra tokens per chunk — usually a good trade for large corpora with repetitive language.

Tables, code and scanned pages

Fixed-size splitters do genuine damage here. Tables: a row without its header row is data with no labels — keep tables whole, and if one must split, repeat the header in each piece. Code: half a function is noise; split at function/class boundaries. Scanned PDFs: if OCR produced garbled text, no chunking strategy saves you — detect it at ingestion (a document that yields zero or nonsense chunks) and fix the extraction instead.

Selection-round radar: "What chunk size would you use?" is a trap for anyone who answers with a single number. Answer with the reasoning: start ~200–500 tokens with overlap, split on the document's own structure where it exists, keep tables and code whole, and tune against a golden set. The word "measure" is what earns the mark.

Common mistakes

• Fixed-size splitting through a document that has clear headings — free structure thrown away.
• Zero overlap, then wondering why one specific fact is unfindable.
• Chunks that lose their heading, so "12 days" belongs to no policy.
• Splitting tables and code blocks mid-structure.
• Copying a chunk size from a tutorial and never measuring it on your own corpus.
• Re-chunking with new settings but forgetting to re-embed and clear the old chunks.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and two tasks — write a splitter with working overlap, then choose a strategy for four very different document types.

Q1Chunkingeasy

Why does chunk overlap exist?

Asked in

aTo make the index bigger for better recall statisticsbSo a fact sitting on a boundary isn't split across two chunks and lost from bothcTo reduce embedding costdBecause vector databases require it

Q2Chunkingmedium

What is the practical downside of very LARGE chunks (say 4,000 tokens)?

Asked in

aThey cannot be embedded at allbTheir embedding averages several topics (blurry match), they crowd the prompt with irrelevant text, and they push up cost per querycThey always exceed the vector database's row limitdThey make the LLM answer faster

Q3Chunkingmedium

And the downside of very SMALL chunks (say 50 tokens)?

Asked in

aRetrieval becomes slowerbFragments lose context — 'the notice period is 30 days' without saying for what — and the answer often needs several chunks stitched togethercEmbeddings can't handle short textdThey cost more to embed than large chunks

Q4Chunkingmedium

'Semantic chunking' means:

Asked in

aChunking with a fixed character countbSplitting where the topic actually shifts — detected by comparing embeddings of consecutive sentences — instead of at arbitrary lengthscLetting the LLM rewrite each chunkdStoring chunks in semantic order

Q5Chunkinghard

You're chunking a document full of tables and code blocks. What's the specific danger of a naive fixed-size splitter?

Asked in

aTables cannot be embeddedbIt cuts tables and code mid-structure, producing chunks with headerless rows or half-functions that are meaningless to both retrieval and the LLMcCode always needs a separate databasedNothing — fixed-size splitting is safe for all content

Q6Chunkingmedium

What is 'contextual chunk enrichment' (adding a short document/section summary to each chunk before embedding)?

Asked in

aA way to reduce index sizebA technique where each chunk is prefixed with context about where it came from, so its embedding — and the LLM reading it — knows which document and section it belongs tocCompressing chunks with gzipdTranslating chunks to English

Hands-on tasks:

FAQ

Next lesson: getting the right chunk to the top of the list — Lesson 6: Retrieval Techniques →