FactoryCrux

FactoryCrux is the ingestion pipeline that converts raw content from URLs, files, and document uploads into signed, indexed artefacts ready for retrieval by the Engine. It is the gateway through which all knowledge enters the CueCrux platform, and its design reflects the principle that data quality at ingestion time determines answer quality at query time.

The ingestion process begins with policy-gated workflows that ensure every piece of content is legally and ethically eligible for processing. Before any content is downloaded or parsed, FactoryCrux checks robots.txt directives, evaluates licence compatibility, scans for personally identifiable information, and applies tenant-specific content policies. These gates are not optional shortcuts that can be bypassed for convenience. They are hard requirements enforced at the pipeline level, and their outcomes are recorded in the provenance ledger.

Once content passes the policy gates, FactoryCrux normalises it into a canonical format regardless of its original structure. PDFs, HTML pages, Word documents, spreadsheets, and plain text files all converge into a unified representation that preserves semantic structure while stripping away format-specific noise. This normalisation stage handles character encoding issues, removes boilerplate navigation and advertising content, and preserves meaningful structural elements like headings, tables, and lists.

The normalised content then flows through the semantic chunking pipeline, which splits documents at natural topic boundaries rather than arbitrary token counts. FactoryCrux measures sentence-to-sentence embedding similarity and creates breakpoints where significant topic shifts occur, producing coherent chunks that preserve the argument structure of the original document. Each chunk receives a stable identity computed as a BLAKE3 hash of its normalised content, meaning identical content always produces the same chunk identity regardless of when or how it was ingested.

Chunk identities feed into the deduplication system, which detects exact copies, near-duplicates, and derivative works across the entire corpus. This prevents evidence inflation, where the same information appears multiple times and creates a false impression of broad support for a claim. Duplicates are consolidated under canonical artefact identities while maintaining provenance links to all original sources.

After chunking and deduplication, FactoryCrux coordinates with embedding workers to generate vector representations for each chunk. The embedding lane is selected based on the corpus class and domain, with different lanes offering different trade-offs between cost, speed, and recall quality. Embedding results are stored alongside the chunks and registered in CoreCrux as lifecycle events.

Throughout this entire process, FactoryCrux maintains a comprehensive provenance ledger recording where each piece of content came from, when it was ingested, what policy gates it passed through, how it was chunked, and which embedding lane was used. This provenance trail is essential for CROWN receipt generation, as it allows Engine to trace any answer back through the full content lifecycle to the original source.

FactoryCrux also handles ongoing maintenance of the corpus, detecting when source documents have been updated, removed, or flagged, and triggering re-ingestion workflows that update the corpus while preserving the historical record.