Stateful Multi-Turn RAG: Contextual Retrieval with Conversation History

The Achilles' Heel of Stateless RAG in Dialogues

Retrieval-Augmented Generation (RAG) has become the de facto standard for grounding Large Language Models (LLMs) in factual, external knowledge. The canonical RAG pipeline—embedding a user query, performing a similarity search against a vector store, and feeding the retrieved context to an LLM—is powerful for single-shot Q&A. However, its stateless nature renders it brittle and often useless in real-world conversational applications.

Consider a user interacting with a chatbot designed to answer questions about a new cloud service, 'QuantumLeap'.

Turn 1:

* User: "Tell me about the QuantumLeap compute service."

* RAG System: (Correctly retrieves docs about QuantumLeap) -> "QuantumLeap is a serverless compute platform..."

Turn 2:

* User: "What are its pricing models?"

An out-of-the-box, stateless RAG system will embed the literal query "What are its pricing models?". The pronoun "its" lacks a concrete subject. The vector search will likely return generic documents about pricing or nothing relevant at all, leading to a complete failure of the conversational flow.

This is the core problem we will solve. This article is not an introduction to RAG; it is a deep dive into production-ready patterns for maintaining conversational state, aimed at engineers who have already built a basic RAG system and have hit this exact wall.

We will analyze and implement three primary strategies, moving from a naive baseline to a robust, production-grade solution.

Baseline: The Failing Stateless Implementation

Let's first establish our failing baseline to illustrate the problem concretely. We'll use a simplified structure with mock components. Assume you have a vector_store with a .similarity_search() method and an llm with a .invoke() method.

import uuid
from typing import List, Dict, Any

# --- Mock Components (Replace with your actual implementations) ---
class MockVectorStore:
    def __init__(self):
        self.documents = {
            "doc1": "QuantumLeap is a serverless compute platform designed for high-throughput data processing.",
            "doc2": "The pricing for QuantumLeap is based on a pay-per-use model, costing $0.0001 per million requests.",
            "doc3": "Security for QuantumLeap includes end-to-end encryption and VPC integration.",
            "doc4": "The legacy system, ChronoShift, has a tiered monthly subscription pricing model."
        }

    def similarity_search(self, query: str, k: int = 1) -> List[str]:
        print(f"\n[DEBUG] Vector Store searching for: '{query}'")
        # A very simplistic search for demonstration purposes
        if "quantumleap" in query.lower() and "pricing" in query.lower():
            return [self.documents["doc2"]]
        if "quantumleap" in query.lower() and "security" in query.lower():
            return [self.documents["doc3"]]
        if "quantumleap" in query.lower():
            return [self.documents["doc1"]]
        if "pricing" in query.lower():
            return [self.documents["doc4"]] # Incorrectly fetches legacy info
        return []

class MockLLM:
    def invoke(self, prompt: str) -> str:
        print(f"[DEBUG] LLM received prompt with length: {len(prompt)}")
        # Mock logic to generate a response
        if "QuantumLeap" in prompt and "pricing" in prompt:
            return "The pricing for QuantumLeap is based on a pay-per-use model."
        if "QuantumLeap" in prompt:
            return "QuantumLeap is a serverless compute platform."
        return "I'm sorry, I don't have enough information to answer that."

# --- RAG Core Logic ---
vector_store = MockVectorStore()
llm = MockLLM()

def stateless_rag_chain(query: str) -> str:
    retrieved_docs = vector_store.similarity_search(query)
    context = "\n".join(retrieved_docs)
    
    prompt = f"""
    Context: {context}
    Question: {query}
    Answer:
    """
    
    return llm.invoke(prompt)

# --- Simulating the Conversation ---
chat_history = []

# Turn 1
query1 = "Tell me about the QuantumLeap compute service."
response1 = stateless_rag_chain(query1)
chat_history.append(("user", query1))
chat_history.append(("ai", response1))
print(f"User: {query1}\nAI: {response1}")

# Turn 2
query2 = "What are its pricing models?"
response2 = stateless_rag_chain(query2)
chat_history.append(("user", query2))
chat_history.append(("ai", response2))
print(f"\nUser: {query2}\nAI: {response2}")

Execution Output:

[DEBUG] Vector Store searching for: 'Tell me about the QuantumLeap compute service.'
[DEBUG] LLM received prompt with length: 153
User: Tell me about the QuantumLeap compute service.
AI: QuantumLeap is a serverless compute platform.

[DEBUG] Vector Store searching for: 'What are its pricing models?'
[DEBUG] Vector Store retrieved documents: ['The legacy system, ChronoShift, has a tiered monthly subscription pricing model.']
[DEBUG] LLM received prompt with length: 151
User: What are its pricing models?
AI: I'm sorry, I don't have enough information to answer that.

The failure is clear. The second query, lacking context, retrieves an irrelevant document about a legacy system, and the LLM, lacking the correct context in its prompt, cannot formulate a correct answer. This is the baseline we must improve upon.

Strategy 1: Naive History Concatenation

The most straightforward approach is to simply prepend the entire conversation history to the new query before sending it to the vector store. The idea is that the combined text will contain all the necessary context for the embedding model.

Implementation

We'll modify our chain to accept and use a chat_history object.

# ... (using the same mock components)

def format_history(chat_history: List[tuple]) -> str:
    return "\n".join([f"{role}: {text}" for role, text in chat_history])

def history_concat_rag_chain(query: str, chat_history: List[tuple]) -> str:
    history_str = format_history(chat_history)
    # Combine history and the new query for retrieval
    contextual_query = f"{history_str}\nuser: {query}"
    
    retrieved_docs = vector_store.similarity_search(contextual_query)
    context = "\n".join(retrieved_docs)
    
    prompt = f"""
    Context: {context}
    
    Conversation History:
    {history_str}
    
    Question: {query}
    Answer:
    """
    
    return llm.invoke(prompt)

# --- Simulating the Conversation ---
chat_history = []

# Turn 1
query1 = "Tell me about the QuantumLeap compute service."
# For the first turn, history is empty
response1 = history_concat_rag_chain(query1, chat_history)
chat_history.append(("user", query1))
chat_history.append(("ai", response1))
print(f"User: {query1}\nAI: {response1}")

# Turn 2
query2 = "What are its pricing models?"
response2 = history_concat_rag_chain(query2, chat_history)
chat_history.append(("user", query2))
chat_history.append(("ai", response2))
print(f"\nUser: {query2}\nAI: {response2}")

Execution Output:

[DEBUG] Vector Store searching for: 'user: Tell me about the QuantumLeap compute service.'
[DEBUG] LLM received prompt with length: 159
User: Tell me about the QuantumLeap compute service.
AI: QuantumLeap is a serverless compute platform.

[DEBUG] Vector Store searching for: 'user: Tell me about the QuantumLeap compute service.
ai: QuantumLeap is a serverless compute platform.
user: What are its pricing models?'
[DEBUG] LLM received prompt with length: 326
User: What are its pricing models?
AI: The pricing for QuantumLeap is based on a pay-per-use model.

It works for this simple case. The concatenated string "...QuantumLeap...pricing..." contains enough keywords for the search to succeed.

Analysis and Production Pitfalls

While simple, this pattern is a ticking time bomb in a production environment.

Edge Case: What if the conversation drifts? If the user talks about QuantumLeap for 10 turns and then asks "Now tell me about ChronoShift", the concatenated query will be heavily biased towards QuantumLeap, potentially hindering the retrieval of ChronoShift documents.

Verdict: This approach is a suitable for a proof-of-concept but is not a viable long-term strategy. It introduces significant performance, cost, and accuracy issues as conversations lengthen.

Strategy 2: Conversational History Summarization

A more sophisticated approach is to manage the growing history by periodically summarizing it. Instead of concatenating the raw history, we use an LLM to create a concise summary of the dialogue so far. This summary, combined with the new query, forms the input for the retrieval step.

Implementation

This introduces a preliminary LLM call into our chain, specifically for the summarization task.

# ... (using the same mock components)

class MockSummarizerLLM:
    def invoke(self, prompt: str) -> str:
        print("[DEBUG] Summarizer LLM invoked.")
        # In a real scenario, this would be a fine-tuned or prompt-engineered LLM
        if "QuantumLeap" in prompt:
            return "The user is asking about the QuantumLeap compute service."
        return "The user is having a general conversation."

summarizer_llm = MockSummarizerLLM()

def summarize_history(chat_history: List[tuple]) -> str:
    if not chat_history:
        return ""
    history_str = format_history(chat_history)
    prompt = f"""
    Summarize the following conversation concisely. Focus on the main entities and topics discussed.
    
    Conversation:
    {history_str}
    
    Summary:
    """
    return summarizer_llm.invoke(prompt)

def summary_rag_chain(query: str, chat_history: List[tuple]) -> str:
    summary = summarize_history(chat_history)
    contextual_query = f"{summary}\n{query}"
    
    retrieved_docs = vector_store.similarity_search(contextual_query)
    context = "\n".join(retrieved_docs)
    
    prompt = f"""
    Context: {context}
    
    Conversation Summary:
    {summary}
    
    Question: {query}
    Answer:
    """
    
    return llm.invoke(prompt)

# --- Simulating the Conversation ---
chat_history = []

# Turn 1
query1 = "Tell me about the QuantumLeap compute service."
response1 = summary_rag_chain(query1, chat_history)
chat_history.append(("user", query1))
chat_history.append(("ai", response1))

# Turn 2
query2 = "What are its pricing models?"
response2 = summary_rag_chain(query2, chat_history)
chat_history.append(("user", query2))
chat_history.append(("ai", response2))
print(f"\nUser: {query2}\nAI: {response2}")

Analysis and Performance Considerations

This method directly addresses the context window problem of the concatenation strategy.

Pros:

* Bounded Context: The summary keeps the historical context to a manageable size, preventing token overflow.

* Reduced Noise: A good summary can distill the conversation to its core topics, potentially creating a cleaner signal for the retrieval model compared to a long, raw history.

Cons:

* Increased Latency: The most significant drawback is the introduction of a blocking LLM call at the beginning of every turn. This can substantially increase the perceived latency for the user.

* Information Loss: Summarization is inherently lossy. The summarizer LLM might fail to preserve a subtle but crucial detail from an earlier turn that is needed to understand the current query. The quality of the entire RAG chain is now dependent on the quality of this summarization step.

* Increased Cost: An extra LLM call on every turn adds directly to operational costs.

Performance Optimization:

To mitigate the latency and cost, consider using a smaller, faster, and cheaper model for the summarization task (e.g., GPT-3.5-Turbo, or a fine-tuned open-source model like Llama-3-8B) while using a more powerful model (e.g., GPT-4o) for the final answer generation. The summarization task is less complex and often doesn't require the most powerful model available.

Edge Case: The summarization model needs to be robust. If it generates a vague summary like "The user asked a question", the entire chain will fail. The prompt for the summarizer is a critical piece of engineering in this pattern.

Verdict: A viable strategy, especially for very long, exploratory conversations where a running summary is valuable. However, the latency overhead makes it less suitable for applications requiring near-instantaneous responses.

Strategy 3: Query Rewriting (The Production Standard)

This is the most widely adopted and generally most effective strategy for handling conversational RAG. Instead of modifying the context, we modify the query itself. The goal is to use an LLM to transform the user's (potentially context-dependent) follow-up query into a fully-formed, standalone query that is optimal for retrieval.

* Original Query: "What are its pricing models?"

* Rewritten Query: "What are the pricing models for the QuantumLeap compute service?"

This rewritten query is perfect for a stateless vector store.

Implementation

The key to this approach is the prompt used for the rewriting LLM. It must instruct the model to use the conversation history to resolve ambiguities and co-references in the latest query.

# ... (using the same mock components)

class MockRewriterLLM:
    def invoke(self, prompt: str) -> str:
        print("[DEBUG] Rewriter LLM invoked.")
        if "QuantumLeap" in prompt and "pricing" in prompt:
            return "What are the pricing models for the QuantumLeap compute service?"
        # Handle cases where no rewrite is needed
        if "is already a standalone question" in prompt:
            return prompt.split("Follow-up query:")[-1].strip()
        return "What is QuantumLeap?" # Fallback

rewriter_llm = MockRewriterLLM()

def rewrite_query(query: str, chat_history: List[tuple]) -> str:
    if not chat_history:
        return query

    history_str = format_history(chat_history)
    
    prompt = f"""
    Given the following conversation history and a follow-up user query, 
    rephrase the follow-up query to be a standalone question.
    It should be self-contained and not rely on any context from the chat history.
    If the follow-up query is already a standalone question, return it as is.

    Chat History:
    {history_str}

    Follow-up query: "{query}"

    Standalone query:
    """
    
    return rewriter_llm.invoke(prompt)

def query_rewrite_rag_chain(query: str, chat_history: List[tuple]) -> str:
    rewritten_query = rewrite_query(query, chat_history)
    print(f"[INFO] Original Query: '{query}' | Rewritten Query: '{rewritten_query}'")
    
    retrieved_docs = vector_store.similarity_search(rewritten_query)
    context = "\n".join(retrieved_docs)
    
    # We still pass history to the final LLM for conversational tone
    history_str = format_history(chat_history)
    
    prompt = f"""
    Context: {context}
    
    Conversation History:
    {history_str}
    
    Question: {query} 
    Answer:
    """
    
    return llm.invoke(prompt)

# --- Simulating the Conversation ---
chat_history = []

# Turn 1
query1 = "Tell me about the QuantumLeap compute service."
response1 = query_rewrite_rag_chain(query1, chat_history)
chat_history.append(("user", query1))
chat_history.append(("ai", response1))

# Turn 2
query2 = "What are its pricing models?"
response2 = query_rewrite_rag_chain(query2, chat_history)
chat_history.append(("user", query2))
chat_history.append(("ai", response2))
print(f"\nUser: {query2}\nAI: {response2}")

Execution Output:

[INFO] Original Query: 'Tell me about the QuantumLeap compute service.' | Rewritten Query: 'Tell me about the QuantumLeap compute service.'
[DEBUG] Vector Store searching for: 'Tell me about the QuantumLeap compute service.'
...

[DEBUG] Rewriter LLM invoked.
[INFO] Original Query: 'What are its pricing models?' | Rewritten Query: 'What are the pricing models for the QuantumLeap compute service?'
[DEBUG] Vector Store searching for: 'What are the pricing models for the QuantumLeap compute service?'
[DEBUG] LLM received prompt with length: 326
User: What are its pricing models?
AI: The pricing for QuantumLeap is based on a pay-per-use model.

The output demonstrates the pattern's success. The rewriter correctly formulates a standalone query, which leads to perfect retrieval.

Analysis and Edge Case Handling

Pros:

* High Retrieval Accuracy: A well-formed, standalone query is the ideal input for a vector store. This method typically yields the highest-quality retrieval results.

* Decoupled Components: The retrieval logic remains simple and stateless. All the complexity of handling conversation history is encapsulated within the query transformation step.

* Efficient: The LLM call for rewriting operates on a relatively small amount of text (history + new query) and is generally faster than a full summarization task.

Cons:

* Latency Overhead: Like summarization, it adds an LLM call to the critical path, increasing latency. However, this is often considered a necessary trade-off for accuracy.

* Prompt Sensitivity: The entire system's performance hinges on the quality of the rewriting prompt. A poorly written prompt can lead to nonsensical or inaccurate rewrites.

Critical Edge Case: The Standalone Query

What if the user's next question is already self-contained? For example: "What are the security features of QuantumLeap?". A naive rewriter might still try to merge it with the history, producing a redundant query. The prompt is crucial here. The instruction "If the follow-up query is already a standalone question, return it as is." is vital to prevent unnecessary and potentially harmful transformations. This allows the system to gracefully handle both context-dependent and context-independent queries within the same flow.

Advanced Pattern: Hybrid Retrieval with Rewriting and Metadata Filtering

For the most demanding production systems, we can combine the power of query rewriting with the precision of metadata filtering. In this pattern, we use the LLM not just to rewrite the query for semantic (dense vector) search, but also to extract structured entities that can be used for filtering (sparse keyword search).

Imagine our documents are indexed not just with their text embeddings, but also with metadata like {"service_name": "QuantumLeap", "category": "pricing"}.

Our advanced rewriter can be prompted to output a JSON object:

{
  "rewritten_query": "pricing models for QuantumLeap compute service",
  "filters": {
    "service_name": "QuantumLeap",
    "category": "pricing"
  }
}

This allows the retrieval system to perform a highly precise hybrid search:

This pattern provides the best of both worlds: the broad semantic understanding of dense vectors and the surgical precision of structured filtering. While more complex to implement, it drastically reduces false positives in retrieval and is a hallmark of a mature, production-grade RAG system.

Final Context Management: The Generation Step

Even with perfect retrieval, you still need to construct the final prompt for the generation LLM. This prompt will contain the retrieved documents, the original user query, and some amount of conversation history to maintain conversational tone and flow. The history can still overflow the context window here.

A robust, token-aware truncation strategy is essential:

This can be implemented using a tokenizer like tiktoken to count tokens accurately before making the final API call, ensuring you never exceed the LLM's limit.

Conclusion

Moving from a stateless RAG prototype to a stateful, conversational AI requires a deliberate strategy for managing dialogue history. We've seen how naive concatenation fails under load and how summarization offers a trade-off between context control and latency.

For most production applications, query rewriting stands out as the most balanced and effective pattern. It isolates the complexity, maximizes retrieval accuracy, and, when engineered with robust prompts, can handle a wide variety of conversational turns gracefully.

By adopting the query rewriting pattern and considering advanced techniques like hybrid search and token-aware context management, you can build RAG systems that are not just factually grounded but also truly conversational, providing a seamless and intelligent user experience.