Hybrid Retrieval

Hybrid Retrieval is the dual-path search strategy that gives CueCrux robust recall across the full spectrum of query types, from precise technical lookups to broad conceptual explorations. By combining two fundamentally different approaches to information retrieval and merging their results through late fusion, the system compensates for the weaknesses of each individual method while amplifying their strengths.

The sparse retrieval path uses BM25, a proven term-frequency-based ranking algorithm that excels at exact keyword matching. When a user queries for a specific regulation number, a named chemical compound, or a precise legal citation, BM25 finds documents containing those exact terms with high precision. It is fast, well-understood, and particularly effective for queries where the user knows the exact terminology used in the source documents.

The dense retrieval path uses vector similarity search over chunk embeddings generated by the Embedding Lane Stack. This path captures semantic similarity rather than lexical overlap, meaning it can find relevant content even when the user's query uses completely different words than the source document. A query about worker safety might retrieve documents discussing occupational health and employee protection even if those documents never use the phrase worker safety. This semantic understanding is crucial for natural language queries where vocabulary mismatch between queries and documents is the norm.

Late fusion is the technique used to merge the ranked lists from both retrieval paths. Rather than combining the retrieval methods at the query or index level, CueCrux runs both paths independently and then merges their ranked results using reciprocal rank fusion. This approach is robust to differences in score distributions between the two methods and produces a combined ranking that benefits from both lexical precision and semantic understanding.

The fusion weights are tuneable per corpus class and query intent. For a legal corpus where precise terminology matters, the BM25 weight might be increased to favour exact matches. For a scientific corpus where conceptual similarity is more important, the vector weight might be elevated. Query intent classification can further adjust the weights dynamically. A query that contains specific identifiers or numbers is likely to benefit from stronger BM25 weighting, while an open-ended conceptual question benefits from stronger semantic weighting.

Hybrid Retrieval also integrates with other retrieval enhancement techniques in the CueCrux pipeline. HyDE-generated hypothetical answers are used to improve the dense retrieval path's recall. Proposition extraction creates fine-grained retrieval units that improve claim-level matching. Hierarchical summaries enable retrieval at different abstraction levels. All of these techniques work within the hybrid retrieval framework to deliver the most relevant evidence for every query.

For the CueCrux platform, Hybrid Retrieval ensures that the evidence presented in CROWN receipts is genuinely the most relevant material available in the corpus. The quality of retrieval directly determines the quality of answers, making this capability foundational to the platform's trustworthiness.