Hybrid search and reranking

Two kinds of search that fail differently

The vector search from the previous lesson matches on meaning. It is the right tool when the user's words and the document's words differ but the intent is the same. "Can contractors expense meals?" should find a policy section titled "Reimbursement eligibility for non-employees" even though they share almost no words.

But meaning matching has a blind spot: exact tokens. If a user pastes ERR_AUTH_4017, they do not want a semantically similar authentication overview. They want the one page that contains that exact string. Embedding models smear rare tokens, identifiers, SKUs, version numbers, and legal citations into a fuzzy neighborhood, so the exact match can rank below a vaguely related paragraph.

Keyword search is the opposite. It is precise about tokens and blind to meaning. This is why the two are paired so often: their failure modes barely overlap, so the union of their results misses far less than either one alone.

What keyword search actually does

The keyword side of hybrid retrieval is usually BM25, the scoring function behind engines like Elasticsearch, OpenSearch, and Lucene. It is worth knowing roughly how it ranks, because it explains when it wins.

BM25 scores a chunk against a query using three intuitions. A term that appears more often in a chunk makes that chunk more relevant, but with saturation: the tenth occurrence adds far less than the second, so keyword spam does not dominate. A term that is rare across the whole corpus (inverse document frequency) counts for more, so "the" is near worthless while "4017" is decisive. And a match in a short chunk counts for more than the same match buried in a long one (length normalization).

People call this sparse retrieval: each chunk is represented by the handful of vocabulary terms it actually contains, most of the dimensions being zero. Vector search is dense: every chunk is a few hundred non-zero numbers. Sparse is literal, dense is semantic. Hybrid retrieval just means you run both and combine them.

The hybrid pipeline

The shape is almost always the same: fan out to several retrievers, merge their candidates into one deduplicated pool, then rank that pool with something stronger before you spend any context budget on it.

The point is not to make retrieval fancy. It is to raise recall before the final selection, because a reranker cannot rank a chunk it never receives. If the right evidence is missing from the pool, no amount of clever scoring downstream can save the answer.

Merging scores you cannot compare

Here is the catch that surprises people: a BM25 score of 14.2 and a cosine similarity of 0.81 are not on the same scale, and they are not even in the same units. You cannot just add them. Normalizing each into 0 to 1 helps a little, but the distributions are lumpy and a single outlier can wreck a min-max scaling.

The robust default is Reciprocal Rank Fusion (RRF). It throws away the raw scores entirely and uses only the rank a chunk got in each list. Each chunk earns 1 / (k + rank) from every list it appears in, with k a small constant (60 is the common starting point), and the chunk's final score is the sum across lists. A chunk that both retrievers rank near the top wins; a chunk that one system loved and the other never returned still does fine. Because it only looks at ranks, RRF does not care that the two systems use incompatible score scales, which is exactly why it travels well across different retrievers.

Weighted fusion (normalize, then take a weighted sum, often called alpha-weighting) is the other common choice and lets you dial how much you trust the semantic side versus the keyword side. It can edge out RRF when you tune it per dataset, but it needs that tuning. RRF is the safer thing to ship first.

Reranking: read the query and the chunk together

First-stage retrieval is fast because it compares precomputed representations. Your embedding model is a bi-encoder: it turns the query into one vector and each chunk into another, separately, and compares them by distance. That separation is what makes it scale to millions of chunks, but it is also the weakness. The model commits to a single vector for a chunk before it has ever seen the query.

A reranker is usually a cross-encoder. It feeds the query and one candidate chunk through a transformer together, so every word of the query can attend to every word of the chunk, and outputs a single relevance score. That joint read catches things distance cannot: negation, qualifiers, whether the chunk actually answers the question or just shares vocabulary with it. It is markedly more accurate at ordering the top results.

The price is that you cannot precompute anything. The reranker has to run a forward pass for every query-chunk pair at request time, so it is far too slow to run over the whole corpus. That is the whole reason for the two-stage shape: cheap retrieval narrows millions of chunks down to maybe 50 to 200 candidates, then the expensive reranker reorders just those.

Hard filters versus soft boosts

Some constraints are not ranking preferences at all. Tenant, user permissions, document status, language, and product version are hard filters: they decide whether a chunk is even eligible, and they must be applied before ranking. A highly relevant document the user is not allowed to see is still forbidden, and a reranker that surfaces it is a data leak, not a good result.

Soft boosts are different. "Prefer newer docs" or "rank official sources above forum posts" are preferences that nudge ranking but should not exclude anything outright. The mistake to avoid is collapsing the two. Run hard filters as filters at the index level, ideally inside the same query so they shrink the candidate set early, and keep boosts as adjustments to the score. Conflating them either leaks data (a boost that should have been a filter) or hides good answers (a filter that should have been a boost).

How many chunks should go to the model?

More chunks raise the odds the right evidence is present, but they also cost more tokens, add latency, and give the model more chances to get distracted or to read two chunks that contradict each other. There is no universal number; it depends on chunk size and task shape.

There is also a placement effect worth knowing: models tend to use evidence at the start and end of a long context more reliably than evidence stranded in the middle, the "lost in the middle" pattern. So passing 30 mediocre chunks is often worse than passing the 5 the reranker is most confident about, because the good one can get buried. Retrieve broadly, rerank, dedupe near-identical chunks, then pass only the best evidence that fits a deliberate budget, and put the strongest chunks where the model reads best.

Quick check