Semantic Deduplication

Semantic Deduplication detects near-duplicate content across the CueCrux corpus using approximate nearest neighbour search on chunk embeddings, ensuring that the knowledge base reflects genuine diversity of evidence rather than inflated counts of what is essentially the same information repeated across multiple sources.

Evidence inflation is a subtle but serious problem in retrieval-augmented systems. If the same fact is stated in ten different web pages that all copied from a single original source, a naive retrieval system might present those ten pages as ten independent sources supporting the claim. This creates a false impression of broad consensus when in reality there is only one original source. Semantic Deduplication prevents this by identifying and consolidating duplicate content regardless of how it entered the corpus.

The deduplication system operates at multiple levels. Exact duplicate detection uses BLAKE3 hash comparison to identify chunks with identical content, which is fast and unambiguous. Near-duplicate detection uses approximate nearest neighbour search on chunk embeddings to find paraphrases, where the same information is expressed in different words. Derivative work detection identifies content that is substantially derived from another source, such as summaries, translations, or lightly edited versions of original material.

When duplicates are detected, they are consolidated under a canonical artefact identity. The canonical version is selected based on factors like source authority, publication date, and content completeness. All other versions are linked to the canonical identity as provenance references, maintaining the full record of where the information appeared while ensuring it is counted only once during retrieval and evidence assessment.

This consolidation has direct benefits for answer quality. The Trust Gradient's source diversity metric reflects genuine independent sources rather than echo-chamber repetition. Coverage Labelling accurately identifies when a claim is supported by truly independent evidence versus when it rests on a single source that has been widely copied. And retrieval ranking avoids wasting capacity on redundant results, surfacing a more diverse evidence set for synthesis.

Semantic Deduplication also helps with corpus maintenance. By tracking the duplication patterns in the corpus, the system can identify low-quality sources that primarily repackage existing content without adding original value. This information feeds into the Data Quality Pipeline to improve future sourcing decisions.

The deduplication process runs continuously as new content is ingested through FactoryCrux and also periodically as a batch process across the existing corpus to catch duplicates that were not detected during initial ingestion. Results are logged in the provenance system for transparency.

For the CueCrux platform, Semantic Deduplication is essential to maintaining the integrity of the evidence layer. It ensures that when the platform says an answer is supported by multiple independent sources, that claim is genuinely true.