Data Quality Pipeline

The Data Quality Pipeline is a multi-stage processing system within FactoryCrux that applies eight distinct quality techniques to every piece of content before it is admitted into the CueCrux corpus. This pipeline ensures that the knowledge base maintains high standards of accuracy, legality, privacy, and freshness, recognising that the quality of answers can never exceed the quality of the underlying data.

Deduplication is the first stage, detecting and handling duplicate or near-duplicate content to prevent evidence inflation. The pipeline checks incoming content against the existing corpus using both exact hash matching for identical copies and semantic similarity comparison for paraphrases and derivative works. Duplicates are consolidated under canonical artefact identities, preserving provenance links to all sources while ensuring that the same information is not counted multiple times during retrieval and synthesis.

Language detection identifies the language of incoming content and routes it to the appropriate processing pipeline. CueCrux supports multiple languages, and accurate language detection ensures that content is chunked, embedded, and indexed using language-appropriate models. Misidentified languages would lead to poor embedding quality and degraded retrieval performance.

PII scanning examines content for personally identifiable information that should not be stored or exposed through the platform. The scanner detects names, email addresses, phone numbers, national identification numbers, and other sensitive data patterns. Detected PII is flagged for review and can be automatically redacted before the content enters the corpus, depending on the tenant's configuration.

Licence verification checks that the platform has the legal right to store, process, and display the content. This includes verifying robots.txt compliance for web-sourced content, checking Creative Commons and other licence declarations, and validating that tenant-uploaded content comes with appropriate usage rights. Content that fails licence verification is rejected with a clear explanation of the issue.

Freshness scoring assesses how current the content is based on publication dates, last-modified timestamps, and the rate of change in the content's domain. Stale content receives lower freshness scores, which influence its ranking during retrieval and its contribution to coverage assessments. Content with very low freshness scores may be flagged for re-sourcing or removal.

Format normalisation converts content from its original format into the platform's canonical representation, handling character encoding issues, removing navigation and advertising boilerplate, and preserving meaningful structural elements. Schema validation ensures that the normalised content conforms to the expected structure before it enters the chunking pipeline. Provenance tagging attaches comprehensive metadata to every piece of content, recording its source, ingestion timestamp, pipeline version, and quality scores.

The Data Quality Pipeline is what gives CueCrux users confidence that the evidence behind their answers meets rigorous quality standards. Every quality decision is logged and auditable, creating a transparent record of how the corpus was curated.