Semantic Chunking

Semantic Chunking is the document splitting strategy that produces coherent, topically unified chunks by detecting natural topic boundaries rather than cutting at arbitrary token counts. This approach recognises that the quality of retrieval depends heavily on the quality of the chunks being retrieved, and that a chunk which mixes multiple topics or splits an argument mid-thought will consistently produce poor results.

The chunking process begins by parsing the document into sentences, preserving the natural linguistic units of the text. Each sentence is then embedded using the appropriate model from the Embedding Lane Stack, producing a vector representation that captures its semantic content. The algorithm then measures the similarity between consecutive sentences, computing a rolling similarity score that reveals where the topic flows smoothly and where it shifts abruptly.

Breakpoints are created at positions where the similarity between consecutive sentences drops below a threshold, indicating a significant topic shift. A document discussing healthcare regulation might flow smoothly from HIPAA privacy rules through data handling requirements, with sentences highly similar to their neighbours, before abruptly shifting to discuss clinical trial protocols. The chunking algorithm detects this transition and creates a breakpoint, ensuring that the HIPAA discussion and the clinical trial discussion become separate chunks.

The result is chunks that are coherent units of meaning. Each chunk represents a complete discussion of a single topic or argument, making it much more useful as a retrieval unit than a fixed-size window that might start mid-sentence and end mid-paragraph. When Engine retrieves a semantically chunked passage in response to a query, the user sees a complete, self-contained piece of evidence rather than an arbitrary text fragment.

Chunk sizes are bounded by configurable minimum and maximum limits to prevent pathological cases. Very long topically uniform passages are split at the next best breakpoint once they exceed the maximum size. Very short passages that might be too brief to be useful are merged with their neighbours if the similarity is high enough.

Semantic Chunking interacts with several other CueCrux capabilities. Late Chunking provides an alternative approach that embeds the full document first and then pools token embeddings into chunk vectors, preserving broader context. Proposition Extraction operates downstream of chunking to extract individual factual claims from semantic chunks. Hierarchical Summaries build multi-level abstractions from semantically chunked content.

For the CueCrux platform, Semantic Chunking is a foundational quality investment. The difference between retrieving coherent semantic chunks and arbitrary text windows is the difference between evidence that naturally supports clear answers and evidence that requires extensive post-processing to be useful.