HyDE

HyDE, or Hypothetical Document Embeddings, is a query expansion technique that bridges the vocabulary gap between how users phrase their questions and how source documents express the relevant information. It works by generating a hypothetical answer to the user's query using a language model, then embedding that hypothetical answer to retrieve real evidence from the corpus.

The vocabulary gap is one of the most persistent challenges in information retrieval. A user might ask about the legal requirements for firing an employee, while the relevant regulatory documents discuss termination procedures and employment separation. Despite addressing the same topic, the query and the source use entirely different terminology. Traditional keyword search would struggle with this mismatch, and even standard vector similarity search can miss the connection if the embedding model was not exposed to enough examples linking these vocabularies.

HyDE addresses this by leveraging the language model's broad knowledge to generate a plausible answer that uses the vocabulary and phrasing that source documents are likely to contain. The language model does not need to produce a correct answer. It needs to produce an answer that is written in the style and vocabulary of the documents in the corpus. When this hypothetical answer is embedded and used as a search query, it naturally matches the real documents that discuss the same topic in similar language.

The hypothetical document generation step is fast and lightweight, adding minimal latency to the retrieval pipeline. The language model generates a brief passage that addresses the query as if writing a paragraph for an authoritative document on the topic. This passage is then embedded using the same model and lane as the corpus chunks, ensuring compatible vector representations.

HyDE is particularly effective for queries that are expressed conversationally or colloquially, where the gap between user language and document language is widest. It also helps with highly technical queries where the user may know the general concept but not the specific terminology used in the relevant standards, regulations, or scientific literature.

Within the CueCrux retrieval pipeline, HyDE operates as part of the Hybrid Retrieval system. The hypothetical document embedding is used as an additional dense retrieval path alongside the original query embedding and the BM25 sparse retrieval. The results from all paths are merged through reciprocal rank fusion, with the HyDE path contributing candidates that the original query might have missed.

HyDE integrates with the CROWN receipt system to ensure transparency. When a retrieval result is surfaced primarily through the HyDE path, the receipt records this fact, allowing auditors to understand why a particular piece of evidence was retrieved even when its terminology differs significantly from the original query.

For the CueCrux platform, HyDE significantly improves recall for natural language queries, ensuring that relevant evidence is not missed simply because users and authors chose different words to express the same concept.