Q2: Document Chunking Strategies: Fixed-Size, Recursive, Semantic & Late Chunking - AI Engineering & LLMs (50 Qs)

1. The Chunking Dilemma

In RAG systems, the quality of retrieved context is strictly bounded by document chunking:

Too small: Destroys context and semantic meaning.
Too large: Ingests noise, exceeds context limits, and dilutes vector similarity scores.

scss

Raw PDF/Doc ──► [ Chunking Engine ] ──► [ Embeddings ] ──► [ Vector Store ]
                      │
  ┌───────────────────┼───────────────────┬───────────────────┐
  ▼                   ▼                   ▼                   ▼
Fixed-Size         Recursive           Semantic           Late Chunking
(Fixed token/char) (Paragraph/Header)  (Embedding shifts) (Whole-doc attention)

2. Chunking Methodologies

Strategy	Mechanics	Pros	Cons	Ideal Use Case
Fixed-Size with Overlap	Splitting every $N$ tokens with $M$ token overlap (e.g., 512 tokens, 50 overlap).	Fast, trivial to implement, constant memory.	Splits sentences across boundaries, ignores semantic flow.	Baseline prototypes, uniform unstructured logs.
Recursive Character Splitting	Splits hierarchically by paragraph (`\n\n`), sentence (`\n`), then space ( ).	Preserves sentence and paragraph boundaries cleanly.	Heuristic; does not detect semantic topic shifts.	General prose, technical documentation, articles.
Semantic Chunking	Computes cosine similarity between consecutive sentences; splits where similarity drops below a threshold.	High semantic cohesion within every chunk.	Requires $N$ embedding calls per document during ingestion.	Complex whitepapers, multi-topic meeting transcripts.
Late Chunking	Passes the entire document through long-context embedding model; pools token embeddings into chunks after attention.	Every chunk retains global document attention context.	Requires long-context embedding models (e.g., 8k tokens).	Legal contracts, financial reports where definitions span pages.

3. Production Recursive Chunking Implementation

python

from typing import List

class RecursiveTextChunker:
    def __init__(self, chunk_size: int = 500, chunk_overlap: int = 50, separators: List[str] = None):
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap
        self.separators = separators or ["\n\n", "\n", ". ", " "]

    def split_text(self, text: str) -> List[str]:
        final_chunks: List[str] = []
        # Recursive splitting logic preserving semantic paragraphs
        splits = self._split(text, self.separators)
        
        current_chunk = ""
        for s in splits:
            if len(current_chunk) + len(s) <= self.chunk_size:
                current_chunk += s
            else:
                if current_chunk:
                    final_chunks.append(current_chunk.strip())
                # Overlap logic
                current_chunk = current_chunk[-self.chunk_overlap:] + s if self.chunk_overlap < len(current_chunk) else s
        
        if current_chunk:
            final_chunks.append(current_chunk.strip())
        return final_chunks

    def _split(self, text: str, separators: List[str]) -> List[str]:
        if not separators:
            return list(text)
        sep = separators[0]
        splits = text.split(sep)
        result = []
        for s in splits:
            if len(s) > self.chunk_size and len(separators) > 1:
                result.extend(self._split(s, separators[1:]))
            else:
                result.append(s + sep)
        return result