Text chunking is the practice of cutting a long document into small pieces before storing it in a knowledge base: instead of feeding a whole book, contract or help center to the model, you split it into chunks of a few hundred to a thousand characters, embed each one separately, and at question time retrieve only the most relevant chunks for the model to answer from. It is the least visible step of a RAG pipeline, and the one that most decides whether answers are accurate.
Why not feed the whole document
Sending an entire document into the context has three problems: it may not fit the window, it is expensive when it does, and the model gets distracted hunting for an answer inside mostly irrelevant text. Chunking lets retrieval pick only the relevant passages, cutting both cost and noise. The price is that the document is artificially torn apart, and only the cutting strategy preserves the connections between parts.
Three common strategies
Fixed-size cutting is simplest but can slice a sentence or a table in half. Cutting along paragraphs and headings follows the original structure and suits manuals and policy documents. Semantic cutting splits only when the topic changes, which works better but is harder to implement. Most tools also leave an overlap between neighbouring chunks, so context cut at a boundary reappears at the start of the next chunk instead of losing the answer at the seam.
What breaks when chunks are too small or too large
Too small, and a single chunk lacks complete information: ask about a reimbursement cap and the retrieved chunk may only say that excess amounts need a VP's approval, while the actual figure sits in the previous chunk, leaving the model to answer vaguely or invent a number. Too large, and one chunk mixes several topics, its embedding averages out to resemble none of them, retrieval ranking suffers, and even when retrieved the model must hunt for the key point inside a long passage.
Three misconceptions
First, smaller is not automatically more precise: it buys finer localisation at the cost of context. Second, more overlap is not always better: heavy overlap stores the same content several times, so the top results are all near-duplicates crowding out other relevant chunks. Third, bad answers are not always the model's fault: when a knowledge base answers poorly, revisiting chunk size and strategy usually helps more than switching models.
There is no universal optimum: clause-style and FAQ documents can be cut smaller, while narrative texts need larger chunks that keep heading structure. The only real test is to run genuine questions and check whether the retrieved chunks can support the correct answer on their own.
