Optimizing RAG Pipelines with Hybrid Search and Re-ranking

Beyond Naive RAG: The Necessity of Multi-Stage Retrieval

For any team that has moved a Retrieval-Augmented Generation (RAG) system from a proof-of-concept to a production environment, the limitations of a simple vector-only retrieval pipeline become painfully obvious. While semantic search is powerful, its reliance on dense vectors often fails when queries contain specific, low-frequency keywords, product SKUs, error codes, or domain-specific jargon. The embedding model, trained on general-purpose text, may not map these critical identifiers to a unique, retrievable vector space, leading to irrelevant context and, consequently, confident hallucinations from the Large Language Model (LLM).

This isn't a problem to be solved with a better embedding model alone. It's an architectural flaw. Production-grade RAG demands a more robust retrieval strategy that marries the lexical precision of classic search algorithms with the semantic richness of modern vector search. This article details the architecture and implementation of such a system: a multi-stage pipeline featuring hybrid search and a final re-ranking layer.

We will construct this pipeline from the ground up, focusing on the practical challenges and trade-offs encountered in a real-world deployment. We will not be covering the basics of what RAG is or how to generate embeddings. The assumption is that you are already building these systems and are looking for advanced techniques to elevate their performance from 'demo-worthy' to 'production-reliable'.

Our final architecture will look like this:

graph TD
    A[User Query] --> B{Sparse Retriever (BM25)};
    A --> C{Dense Retriever (FAISS)};
    B --> D[Top-K Sparse Results];
    C --> E[Top-K Dense Results];
    D --> F{Reciprocal Rank Fusion (RRF)};
    E --> F;
    F --> G[Fused Candidate Set];
    G --> H{Cross-Encoder Re-ranker};
    H --> I[Final Top-N Relevant Chunks];
    I --> J{LLM Prompt Construction};
    J --> K[LLM Generation];
    K --> L[Final Answer];

This multi-stage approach systematically filters and refines the context provided to the LLM, ensuring each stage adds a layer of precision, culminating in a final context that is both semantically and lexically relevant to the user's query.

Stage 1: The Dual Retrievers - BM25 and FAISS

The core of our hybrid search system is the parallel operation of two distinct retrieval mechanisms: a sparse retriever for lexical matching and a dense retriever for semantic matching.

The Sparse Retriever: BM25 for Keyword Precision

Okapi BM25 is a bag-of-words retrieval function that ranks documents based on the query terms they contain. It's an evolution of TF-IDF and is ruthlessly effective at finding documents that contain the exact keywords from a query. This is non-negotiable for queries involving identifiers.

Consider a knowledge base of technical documentation. A user query like "Fix for error code 0x80070005 in MSVC 14.2" will likely fail in a pure vector search system if the embedding model hasn't seen that specific error code frequently. BM25, however, will instantly surface documents containing that exact string.

Implementation with `rank-bm25`

Let's implement a simple BM25 retriever. We'll use the rank-bm25 library, a lightweight Python implementation.

First, let's define our sample corpus. Notice the mix of semantic concepts and specific identifiers.

import numpy as np

# Sample corpus of documents
docs = [
    "The new X-Fusion Pro laptop features a 12th-gen Intel Core i9 processor and 32GB of RAM.",
    "To resolve error 503, you must restart the primary application server.",
    "Our quantum computing framework, 'Quasar', is built on principles of superposition.",
    "The recommended memory for the X-Fusion Pro is DDR5 at 4800MHz.",
    "Server error 503 indicates a service is temporarily unavailable.",
    "Quantum entanglement is a key feature of the Quasar system.",
    "Users experiencing login issues with product SKU 'XFP-2023' should clear their browser cache.",
    "The Intel Core i9-12900K is the flagship CPU in the 12th generation lineup."
]

# Pre-processing: simple tokenization
tokenized_corpus = [doc.lower().split(" ") for doc in docs]

Now, we can set up our BM25 index.

from rank_bm25 import BM25Okapi

bm25 = BM25Okapi(tokenized_corpus)

def get_bm25_results(query, k=5):
    tokenized_query = query.lower().split(" ")
    doc_scores = bm25.get_scores(tokenized_query)
    
    # Get top_k indices and scores
    top_k_indices = np.argsort(doc_scores)[::-1][:k]
    top_k_scores = doc_scores[top_k_indices]
    
    # Return a list of (doc_id, score) tuples
    return list(zip(top_k_indices, top_k_scores))

# Example query that benefits from lexical search
query_lexical = "troubleshoot error 503"
results_lexical = get_bm25_results(query_lexical)

print(f"BM25 Results for: '{query_lexical}'")
for doc_id, score in results_lexical:
    if score > 0:
        print(f"  Score: {score:.2f}, Doc: {docs[doc_id]}")

# Example query with specific SKU
query_sku = "XFP-2023 login"
results_sku = get_bm25_results(query_sku)

print(f"\nBM25 Results for: '{query_sku}'")
for doc_id, score in results_sku:
    if score > 0:
        print(f"  Score: {score:.2f}, Doc: {docs[doc_id]}")

This demonstrates the power of BM25. It correctly identifies documents with the exact terms "503" and "XFP-2023", even if the surrounding context is different.

The Dense Retriever: FAISS for Semantic Understanding

For capturing semantic meaning, we rely on a dense retriever. This involves encoding our documents and query into high-dimensional vectors and finding the nearest neighbors in that vector space. We'll use the sentence-transformers library for embeddings and faiss-cpu for efficient similarity search.

Implementation with `sentence-transformers` and `faiss`

import faiss
from sentence_transformers import SentenceTransformer

# 1. Initialize the embedding model
# 'all-MiniLM-L6-v2' is a good starting point: fast and reasonably powerful.
model = SentenceTransformer('all-MiniLM-L6-v2')

# 2. Encode the documents
doc_embeddings = model.encode(docs, convert_to_tensor=False)

# 3. Build the FAISS index
dimension = doc_embeddings.shape[1]
index = faiss.IndexFlatL2(dimension) # Using L2 distance
index.add(doc_embeddings)

def get_faiss_results(query, k=5):
    query_embedding = model.encode([query])
    distances, indices = index.search(query_embedding, k)
    
    # FAISS returns L2 distances. We can convert them to a similarity score (0-1 range)
    # This is a simple heuristic, more advanced normalization might be needed.
    scores = 1 / (1 + distances[0])
    
    return list(zip(indices[0], scores))

# Example query that benefits from semantic search
query_semantic = "computer hardware specs"
results_semantic = get_faiss_results(query_semantic)

print(f"FAISS Results for: '{query_semantic}'")
for doc_id, score in results_semantic:
    print(f"  Score: {score:.2f}, Doc: {docs[doc_id]}")

Notice that for the query "computer hardware specs", FAISS correctly identifies documents about the "X-Fusion Pro laptop" and its components, even though the exact keywords "computer", "hardware", or "specs" are not present in all of them. This is where a pure BM25 system would fail.

Stage 2: Fusing the Results with Reciprocal Rank Fusion (RRF)

Now we have two ranked lists of documents, one from BM25 and one from FAISS. How do we combine them into a single, superior list? A naive approach would be to normalize the scores and add them up. This is problematic because the score scales from BM25 and FAISS are entirely different and not directly comparable. BM25 scores are unbounded, while our FAISS similarity is in the [0, 1] range.

This is where Reciprocal Rank Fusion (RRF) comes in. RRF is a simple yet incredibly effective technique that disregards the absolute scores and instead uses the rank of each document in the lists. The formula is:

RRF_Score(d) = Σ (1 / (k + rank_i(d))) for each list i

Where rank_i(d) is the rank of document d in list i, and k is a constant (usually set to 60) that dampens the influence of lower-ranked items.

This approach has several advantages:

RRF Implementation

Let's write a function to perform RRF on our two result sets.

from collections import defaultdict

def reciprocal_rank_fusion(list_of_results, k=60):
    # list_of_results is a list where each element is another list of (doc_id, score) tuples
    
    # Use a defaultdict to store the RRF scores for each document
    rrf_scores = defaultdict(float)
    
    # Iterate through each result list (e.g., from BM25, FAISS)
    for results in list_of_results:
        # Iterate through the ranked documents in the current list
        for rank, (doc_id, _) in enumerate(results):
            rrf_scores[doc_id] += 1 / (k + rank + 1) # rank is 0-indexed
            
    # Sort the documents by their final RRF score in descending order
    sorted_docs = sorted(rrf_scores.items(), key=lambda item: item[1], reverse=True)
    
    return sorted_docs

# Let's test with a query that needs both lexical and semantic understanding
query_hybrid = "information about the Quasar i9 processor"

bm25_res = get_bm25_results(query_hybrid, k=5)
faiss_res = get_faiss_results(query_hybrid, k=5)

print("--- Individual Retriever Results ---")
print(f"BM25 Top 3: {[docs[i] for i, s in bm25_res[:3] if s > 0]}")
print(f"FAISS Top 3: {[docs[i] for i, s in faiss_res[:3]]}")

# Now, fuse them
fused_results = reciprocal_rank_fusion([bm25_res, faiss_res])

print("\n--- Fused Results (RRF) ---")
for doc_id, score in fused_results:
    print(f"  Score: {score:.4f}, Doc: {docs[doc_id]}")

For the query "information about the Quasar i9 processor", BM25 will latch onto "i9" and "processor", while FAISS will understand the semantic link between "Quasar" and "quantum computing". The RRF result successfully brings the most relevant documents—those mentioning both concepts—to the very top, something neither retriever could do alone.

Stage 3: The Re-ranker - Precision with Cross-Encoders

Hybrid search gives us a set of highly relevant candidate documents. However, the retrieval process is still based on evaluating the query and documents independently (or via a simple dot-product for vectors). To achieve the highest level of precision, we need a model that can perform a deep, contextual comparison of the query against each candidate document.

This is the role of a Cross-Encoder. Unlike the Bi-Encoder (like our SentenceTransformer) which creates separate embeddings for the query and document, a Cross-Encoder takes both the query and a document as a single input and outputs a relevance score.

Bi-Encoder: score = similarity(encode(query), encode(doc))

Cross-Encoder: score = model([query, doc])

This architectural difference allows the cross-encoder to pay full attention to the interactions between the query and document tokens, making it significantly more accurate. The trade-off is speed; it's far too slow to run on an entire corpus, but it's perfect for re-ranking a small set of promising candidates (e.g., the top 20-50 from our RRF stage).

Implementation with `sentence-transformers` Cross-Encoder

We will use a pre-trained cross-encoder model designed for this task, like ms-marco-MiniLM-L-6-v2.

from sentence_transformers.cross_encoder import CrossEncoder

# Initialize the cross-encoder model
cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def rerank_with_cross_encoder(query, doc_ids):
    # Create pairs of [query, document_text] for the cross-encoder
    query_doc_pairs = [[query, docs[doc_id]] for doc_id in doc_ids]
    
    # Get the scores from the model
    scores = cross_encoder.predict(query_doc_pairs)
    
    # Combine the doc_ids with their new scores
    reranked_results = list(zip(doc_ids, scores))
    
    # Sort by the new score in descending order
    reranked_results.sort(key=lambda x: x[1], reverse=True)
    
    return reranked_results

# Let's use the results from our previous hybrid query
# We take the document IDs from the RRF fused list
fused_doc_ids = [doc_id for doc_id, score in fused_results]

# Now, re-rank these candidates
reranked_results = rerank_with_cross_encoder(query_hybrid, fused_doc_ids)

print("--- Re-ranked Results (Cross-Encoder) ---")
for doc_id, score in reranked_results:
    print(f"  Score: {score:.4f}, Doc: {docs[doc_id]}")

The output will show a refined ordering of the documents, with scores that more accurately reflect the true relevance of each document to the query. The cross-encoder can disambiguate subtle cases that the initial retrievers might miss. For example, it can better determine if a document mentioning "i9 processor" is actually talking about the specific one relevant to the "Quasar" context.

The Complete Production-Grade Pipeline

Now, let's encapsulate this entire logic into a single, reusable class that represents our advanced RAG retrieval engine.

# (Assuming all previous imports and setup: docs, bm25, model, index, cross_encoder)

class AdvancedRAGPipeline:
    def __init__(self, docs, model_name='all-MiniLM-L6-v2', cross_encoder_name='cross-encoder/ms-marco-MiniLM-L-6-v2'):
        self.docs = docs
        self.model = SentenceTransformer(model_name)
        self.cross_encoder = CrossEncoder(cross_encoder_name)
        
        # Setup BM25
        tokenized_corpus = [doc.lower().split(" ") for doc in self.docs]
        self.bm25 = BM25Okapi(tokenized_corpus)
        
        # Setup FAISS
        doc_embeddings = self.model.encode(self.docs, convert_to_tensor=False)
        dimension = doc_embeddings.shape[1]
        self.index = faiss.IndexFlatL2(dimension)
        self.index.add(doc_embeddings)

    def retrieve(self, query, k_sparse=10, k_dense=10, k_rerank=5):
        # 1. Sparse Retrieval (BM25)
        tokenized_query = query.lower().split(" ")
        bm25_scores = self.bm25.get_scores(tokenized_query)
        top_k_bm25_indices = np.argsort(bm25_scores)[::-1][:k_sparse]
        bm25_results = [(idx, bm25_scores[idx]) for idx in top_k_bm25_indices if bm25_scores[idx] > 0]
        
        # 2. Dense Retrieval (FAISS)
        query_embedding = self.model.encode([query])
        distances, indices = self.index.search(query_embedding, k_dense)
        faiss_scores = 1 / (1 + distances[0])
        faiss_results = list(zip(indices[0], faiss_scores))
        
        # 3. Reciprocal Rank Fusion
        fused_results = self.reciprocal_rank_fusion([bm25_results, faiss_results])
        
        # If no results after fusion, return empty
        if not fused_results:
            return []
            
        # 4. Cross-Encoder Re-ranking
        fused_doc_ids = [doc_id for doc_id, score in fused_results[:k_rerank]]
        
        query_doc_pairs = [[query, self.docs[doc_id]] for doc_id in fused_doc_ids]
        reranker_scores = self.cross_encoder.predict(query_doc_pairs)
        
        reranked_results = list(zip(fused_doc_ids, reranker_scores))
        reranked_results.sort(key=lambda x: x[1], reverse=True)
        
        # 5. Return final documents and scores
        final_results = [{'doc': self.docs[doc_id], 'score': float(score), 'id': int(doc_id)} for doc_id, score in reranked_results]
        return final_results

    def reciprocal_rank_fusion(self, list_of_results, k=60):
        rrf_scores = defaultdict(float)
        for results in list_of_results:
            for rank, (doc_id, _) in enumerate(results):
                rrf_scores[doc_id] += 1 / (k + rank + 1)
        return sorted(rrf_scores.items(), key=lambda item: item[1], reverse=True)

# --- Usage ---

pipeline = AdvancedRAGPipeline(docs)

final_query = "What CPU is in the X-Fusion Pro laptop?"
final_context = pipeline.retrieve(final_query, k_rerank=10)

print(f"Query: {final_query}\n")
print("--- Final Retrieved Context for LLM ---")
for item in final_context:
    print(f"  ID: {item['id']}, Score: {item['score']:.4f}, Doc: {item['doc']}")

# This context would then be formatted into a prompt for the LLM.
# For example:
# CONTEXT:
# - The new X-Fusion Pro laptop features a 12th-gen Intel Core i9 processor and 32GB of RAM.
# - The Intel Core i9-12900K is the flagship CPU in the 12th generation lineup.
# ...
# QUESTION: What CPU is in the X-Fusion Pro laptop?
# ANSWER:

This class provides a clean interface to our sophisticated retrieval logic. The k_sparse, k_dense, and k_rerank parameters are critical tuning knobs for balancing performance and accuracy.

Advanced Considerations and Production Edge Cases

Deploying this system at scale introduces further challenges.

* Tuning k values: The most critical optimization. k_sparse and k_dense determine the number of candidates for fusion, while k_rerank directly impacts the most expensive step. A common pattern is to retrieve more candidates initially (e.g., k_sparse=50, k_dense=50) and then re-rank a smaller subset (e.g., k_rerank=10). This requires empirical testing on your specific dataset and latency budget.

* Hardware Acceleration: The embedding and re-ranking stages are prime candidates for GPU acceleration. For high-throughput systems, hosting the SentenceTransformer and CrossEncoder models on a dedicated inference server (like Triton Inference Server) with GPUs is essential.

* Model Quantization: Techniques like ONNX runtime and quantization can significantly reduce model size and speed up inference on both CPUs and GPUs, often with a negligible impact on accuracy.

* Sparse Index: Replace rank-bm25 with a production-grade search engine like Elasticsearch or OpenSearch, which provide distributed, scalable BM25 implementations out of the box.

* Dense Index: Replace faiss.IndexFlatL2 with a managed vector database like Pinecone, Weaviate, Milvus, or Qdrant. These services handle sharding, replication, and efficient indexing (using algorithms like HNSW) for billions of vectors.

* The core pipeline logic remains the same. You simply replace the local get_bm25_results and get_faiss_results calls with API calls to these external services.

* Chunk Size: BM25 tends to favor larger chunks with more keyword overlap, while vector search often performs better with smaller, more semantically coherent chunks. This is a fundamental tension.

* Solution: Multi-layered Context: A powerful pattern is to embed smaller chunks (e.g., sentences or small paragraphs) for retrieval but store metadata linking them back to a larger parent document. When a small chunk is retrieved, you can provide the LLM with the surrounding sentences or the full parent document as context, giving it the best of both worlds.

Conclusion

Moving beyond naive RAG is a necessary step for building reliable, production-ready AI applications. By architecting a multi-stage retrieval pipeline that leverages the strengths of both sparse and dense search and refines the results with a powerful cross-encoder re-ranker, we can drastically improve retrieval relevance. This directly translates to more accurate, less hallucinatory responses from the LLM and a more trustworthy system overall.

The added complexity is not trivial. It requires careful tuning, a robust infrastructure, and a deep understanding of the trade-offs between latency, cost, and accuracy. However, for applications where the quality of the generated output is paramount, this investment in a sophisticated retrieval architecture is not just beneficial—it's essential.