How does document ingestion actually work?

When you upload a document to CueCrux, it goes through a multi-stage ingestion pipeline that transforms a raw file into searchable, receipted knowledge. Understanding this process helps you appreciate why CueCrux's answers from your documents are so precise and traceable.

The first stage is format detection and extraction. CueCrux identifies the file type and applies the appropriate extraction method. For text-based PDFs, it pulls out the raw text along with positional information so it knows where on the page each piece of text appears. For scanned documents, it runs OCR. For Word documents, it parses the XML structure. For spreadsheets, it reads the cell data and formulas. This stage produces a clean text representation of the document along with structural metadata like headings, paragraphs, tables, lists, and page numbers.

The second stage is structural analysis. CueCrux does not just treat your document as a flat stream of text. It analyses the structure to understand how the document is organised. It identifies sections and subsections based on heading levels. It recognises tables and preserves their row-column relationships. It identifies lists and numbered items. It detects footnotes and endnotes and links them to the text they annotate. It finds cross-references within the document. This structural understanding is critical because it means CueCrux can answer questions like "What does Section 4.2 say about liability?" or "What are the items in Table 3?" without losing context.

The third stage is semantic chunking. The document is broken into chunks that are meaningful units of information rather than arbitrary fixed-length segments. CueCrux uses a combination of structural cues and semantic analysis to decide where to break the text. A chunk might be a paragraph, a table, a list, a section, or some other logically coherent piece of content. The goal is that each chunk can stand on its own as a piece of evidence. Chunks overlap slightly at their boundaries to ensure that information that spans a boundary is not lost.

The fourth stage is embedding generation. Each chunk is converted into a high-dimensional vector representation using a transformer-based embedding model. These vectors capture the semantic meaning of the text, not just the keywords. This is what enables CueCrux to find relevant information even when you phrase your question differently from how the document phrases the answer. Two chunks that say the same thing in different words will have similar vectors, so a search for one will find the other.

The fifth stage is fingerprinting and provenance registration. Every chunk gets a cryptographic fingerprint using BLAKE3 hashing. This fingerprint is unique to that exact piece of text from that exact document. It is registered in CueCrux's provenance ledger along with metadata about when the document was uploaded, who uploaded it, and which workspace it belongs to. This is the foundation of the receipt system. When a chunk is later used to answer a question, the receipt can point back to this fingerprint as proof of where the evidence came from.

The sixth stage is indexing. The chunks, their embeddings, their structural metadata, and their fingerprints are all loaded into CueCrux's search infrastructure. This includes both a traditional keyword index for exact-match searches and a vector index for semantic searches. The dual-index approach means that CueCrux can find information whether you search by exact terms or by meaning.

The seventh and final stage is quality assessment. CueCrux evaluates the overall quality of the ingested document. It checks for things like OCR confidence scores, text coherence, structural completeness, and language detection. If there are quality issues, they are flagged so you know which parts of the document might produce less reliable results. For example, if a scanned page had poor image quality and the OCR confidence is low, CueCrux will note this and weight that chunk lower in search results.

The entire pipeline is designed to be fast, accurate, and transparent. Most documents complete ingestion within a few minutes, and you can track the progress in real time. Once ingestion is complete, the document is immediately available for searching and its content will be included in future answers with full receipt tracing.