How vector search works: similarity, top-k and hybrid search

In a RAG system, the retrieval step answers one question: which chunks should the model see? Vector search is the most common answer. This article walks through how it works, from a single similarity score to the hybrid ranking used by most serious systems.

Step 1: score every chunk

Once every chunk has an embedding, searching is conceptually simple:

1. Embed the query with the same model.
2. Compute the similarity between the query vector and each chunk vector.
3. Sort and return the best k.

import numpy as np
 
def top_k(query_vec, chunk_vecs, k=5):
    # vectors are assumed normalised, so the dot product is cosine similarity
    scores = chunk_vecs @ query_vec
    best = np.argsort(-scores)[:k]
    return [(int(i), float(scores[i])) for i in best]

This is exact nearest-neighbour search. With a few thousand chunks it takes milliseconds.

Step 2: make it fast at scale

With millions of vectors, comparing against every one is too slow for interactive use. Approximate nearest neighbour (ANN) indexes trade a little accuracy for a lot of speed:

• HNSW builds a layered graph where each vector links to nearby vectors; search hops through the graph towards the query.
• IVF clusters vectors and only searches the clusters closest to the query.
• Quantisation stores compressed vectors to save memory, at some cost in precision.

Vector databases and libraries such as FAISS implement these. The important thing to know: ANN results can occasionally miss a true nearest neighbour, and index settings trade speed against that recall.

Step 3: choose k and a threshold

• k too small: the right chunk is ranked 6th and never reaches the model.
• k too large: the prompt fills with loosely related text, which costs tokens and can confuse the answer.
• No threshold: the system always returns something, even for questions your documents do not cover.

A minimum score threshold lets the system notice "nothing relevant was found" and say so honestly instead of answering from thin evidence. Similarity scores are not calibrated probabilities, so pick the threshold by looking at scores on real questions.

Keyword search still matters: BM25

BM25 is a classic ranking function used by search engines for decades. It scores a document higher when:

• it contains the query's words,
• those words are rare across the collection (inverse document frequency), and
• the document is not padded out by length (length normalisation).

BM25 excels exactly where embeddings are weak: exact identifiers such as CVE-2024-3094, ERR_CONNECTION_RESET, port numbers, function names and acronyms.

Hybrid search and reciprocal rank fusion

Hybrid search runs both searches and merges the results. Raw BM25 scores and cosine scores are on different scales, so instead of adding scores, a popular method merges ranks. Reciprocal rank fusion (RRF), described by Cormack and colleagues in 2009, gives each item:

RRF(d) = sum over each ranked list of  1 / (k + rank_of_d_in_that_list)

with k commonly set to 60. An item ranked first in either list gets a strong score; an item ranked well in both gets the strongest.

def rrf(ranked_lists, k=60):
    scores = {}
    for ranking in ranked_lists:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)
 
print(rrf([["a", "b", "c"], ["c", "a", "d"]]))  # ['a', 'c', 'b', 'd']

Optional step 4: rerank

A reranker is a model that looks at the query and one candidate chunk together and outputs a relevance score. It is slower than vector comparison but more accurate, so a common design is: retrieve the top 20 to 50 cheaply, rerank them, and keep the best 3 to 5 for the prompt.

Summary

Start simple, measure recall on real questions, and add complexity only when the numbers show you need it. The next article covers how to evaluate a RAG system.