Your RAG chatbot demos beautifully. Then users start asking real questions and it either returns the wrong document, ignores the retrieved context, or hallucinates an answer from scratch. Here is why this happens and what to change.
The Core Problem: Retrieval Quality
RAG stands or falls on retrieval. The LLM cannot answer correctly if the right context is not in its prompt. Most "bad AI answers" are actually bad retrieval problems in disguise.
1. Your Chunk Size Is Wrong for Your Documents
Chunking is the most consequential parameter in a RAG pipeline and the most commonly left at its default value.
Too large a chunk (>1000 tokens): The retrieved chunk contains the answer buried in unrelated text. The LLM gets confused by the noise.
Too small a chunk (<100 tokens): The chunk contains only part of the answer. Critical context lives in the adjacent chunk that was not retrieved.
The right chunk size depends on your document type:
| Document type | Recommended chunk size | |---|---| | FAQ / structured Q&A | 100-200 tokens | | Technical documentation | 300-500 tokens | | Legal / policy documents | 500-800 tokens | | Research papers | 800-1200 tokens |
Add overlap between chunks to avoid losing context at boundaries:
from langchain.text_splitter import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter( chunk_size=500, # tokens chunk_overlap=50, # tokens of overlap between chunks separators=["\n\n", "\n", ". ", " ", ""] # split at paragraph first, then sentence ) docs = splitter.split_documents(raw_docs)
2. The Embedding Model Does Not Match Your Domain
text-embedding-ada-002 and similar general-purpose embeddings work well for general text. They work poorly for specialised domains — legal, medical, code, financial documents — where terminology has precise meanings that a general embedding model maps poorly.
Symptom: Semantically similar queries return different results. "How do I cancel my subscription?" and "What is the cancellation policy?" should retrieve the same document — but they might not.
Fix: Use a domain-appropriate embedding model, or fine-tune embeddings on your corpus.
# For code documentation from langchain.embeddings import HuggingFaceEmbeddings embeddings = HuggingFaceEmbeddings( model_name="microsoft/codebert-base" # better for code docs ) # For general but higher quality embeddings = HuggingFaceEmbeddings( model_name="BAAI/bge-large-en-v1.5" # consistently outperforms ada-002 on MTEB )
3. No Reranking After Retrieval
Vector search retrieves the top-k semantically similar chunks. But semantic similarity is not the same as relevance to a specific question. The fourth or fifth result is sometimes more relevant than the first.
Reranking passes the retrieved chunks through a cross-encoder model that scores each one against the query directly — much more accurate than cosine similarity, but too slow to run against the entire document corpus.
from langchain.retrievers import ContextualCompressionRetriever from langchain.retrievers.document_compressors import CohereRerank # Retrieve more candidates, rerank to top 3 base_retriever = vectorstore.as_retriever(search_kwargs={"k": 20}) compressor = CohereRerank(top_n=3) # or use cross-encoder locally compression_retriever = ContextualCompressionRetriever( base_compressor=compressor, base_retriever=base_retriever ) docs = compression_retriever.get_relevant_documents(query)
Retrieval accuracy typically improves 15-30% with reranking. It is one of the highest-return optimisations in a RAG pipeline.
4. The LLM Is Ignoring the Context
You are retrieving the right documents, but the LLM answers from its training data instead. This happens when:
- ▸The prompt does not clearly instruct the model to use the provided context
- ▸The context is too long and the answer is buried (lost in the middle problem)
- ▸The model's training data contains a contradictory answer it considers more authoritative
Fix the prompt:
system_prompt = """You are a helpful assistant. Answer the user's question using ONLY the information provided in the context below. If the answer is not found in the context, say "I don't have that information in the provided documents" — do not use your general knowledge. Context: {context} """
The phrase "using ONLY the information provided" is important. Without it, models default to their training data for familiar topics.
Fix long context: Put the most relevant chunk first. Research shows LLMs recall information at the beginning and end of context better than the middle.
5. You Are Not Preprocessing the Query
Users do not write retrieval queries. They write conversational questions. "What's the thing about the refund?" is a terrible retrieval query — but a perfectly natural user message.
Query expansion: Rewrite the user query into a better retrieval query before embedding it.
from langchain.chat_models import ChatOpenAI llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0) def expand_query(user_question: str) -> str: prompt = f"""Rewrite the following user question as a clear, specific search query that will retrieve relevant documents from a knowledge base. Remove conversational language. Focus on the key concepts. User question: {user_question} Search query:""" return llm.predict(prompt) query = expand_query("What's the thing about the refund?") # → "refund policy cancellation terms" docs = retriever.get_relevant_documents(query)
Multi-query retrieval: Generate multiple phrasings of the same question and take the union of results.
from langchain.retrievers.multi_query import MultiQueryRetriever multi_retriever = MultiQueryRetriever.from_llm( retriever=base_retriever, llm=llm ) # Generates 3-5 query variants, deduplicates results docs = multi_retriever.get_relevant_documents(question)
Where to Start
If your RAG pipeline is returning wrong answers, diagnose in this order:
- ▸Log the retrieved chunks alongside the answer — are the right documents being retrieved?
- ▸If not: fix chunk size and overlap first, then try reranking
- ▸If yes but the answer is still wrong: fix the prompt and check for lost-in-middle issues
- ▸If retrieval is borderline: add query preprocessing and multi-query retrieval
Fix retrieval before touching the LLM. Nine times out of ten, the problem is in the retrieval stage.