Chunking

Chunking splits text into smaller segments using strategies such as fixed token counts, sentence boundaries, paragraph breaks, or recursive separators. Each chunk is embedded and indexed separately so retrieval returns the most relevant segment rather than an entire document.

That division has real consequences for retrieval quality. Too small and context is lost; a chunk containing only half a sentence may miss the point. Too large and the embedding becomes diluted; a five-thousand-token chunk mixes the relevant sentence with unrelated paragraphs. Chunking strategy determines how well semantic search actually works in practice.

Think of it like this. Think of tearing a book into pages before indexing. If pages are too short, they make no sense alone. If they are too long, the indexer cannot tell which part matters. The right page size makes the book searchable.

How it works

Text is split by character count, token count, sentence, paragraph, or semantic similarity. Overlap between chunks preserves context at boundaries. Recursive splitters try progressively smaller separators to avoid breaking mid-thought. Some systems use content-aware chunking based on document structure.

Common mistakes

"Larger chunks preserve more context." They also dilute the embedding and return irrelevant passages. "Chunking does not matter." It is one of the biggest levers on retrieval quality. "One size fits all." Legal documents, code, and prose need different strategies.

Trade-offs

Small chunks return precise matches but lose context; large chunks preserve context but introduce noise. The right size depends on query type, document structure, and the embedding model's effective receptive field.