Late Chunking

Late Chunking is an advanced embedding strategy that preserves cross-sentence and cross-paragraph context that traditional chunking approaches inevitably lose. Instead of first splitting a document into chunks and then embedding each chunk independently, Late Chunking reverses the order: it embeds the entire document through a long-context model first, and then pools the token-level embeddings into chunk-level vectors after the full document context has been encoded.

The fundamental insight behind Late Chunking is that the meaning of a passage often depends on context that appears elsewhere in the document. A paragraph deep in a legal contract might reference definitions from the preamble, exceptions from an earlier section, or conditions from a subsequent clause. When this paragraph is embedded in isolation, those contextual dependencies are invisible to the embedding model, producing a vector that captures only the local meaning. Late Chunking solves this by letting the transformer model process the entire document, allowing attention mechanisms to establish connections across all positions before the embeddings are pooled into chunks.

The pooling step uses semantic chunking boundaries to determine where chunks begin and end. Token embeddings within each chunk boundary are aggregated using a pooling strategy, typically mean pooling, to produce a single vector for each chunk. Because these token embeddings were produced in the context of the full document, the resulting chunk vectors carry information about the broader context that would be absent in independently embedded chunks.

Late Chunking is particularly valuable for documents with complex internal structure, such as legal contracts, regulatory frameworks, scientific papers with methods and results sections, and technical specifications with cross-references. In these documents, the meaning of any given section is deeply intertwined with the rest of the document, and Late Chunking captures these interdependencies.

The trade-off is computational cost. Processing entire documents through a long-context transformer model requires more GPU memory and time than embedding short chunks independently. This is why Late Chunking is typically assigned to the premium or specialist lanes in the Embedding Lane Stack, reserved for high-value content where the quality improvement justifies the additional cost.

Within the CueCrux retrieval pipeline, Late Chunking complements Semantic Chunking rather than replacing it. Semantic Chunking determines where the chunk boundaries fall based on topic shifts, while Late Chunking determines how those chunks are embedded. The combination produces coherent, topically unified chunks with context-aware vector representations.

Late Chunking results are tracked in the lane metadata recorded in CROWN receipts, ensuring that auditors can see which chunks benefited from full-document context during embedding and assess whether the embedding approach was appropriate for the query.

For the CueCrux platform, Late Chunking pushes retrieval quality beyond what is achievable with conventional chunk-and-embed approaches, particularly for the complex, professionally authored documents that are most important in regulated and professional domains.