Your document Q&A bot demos beautifully. You ask it the questions you prepared, it answers them well, everyone nods. Then you put it in front of real users, they ask real questions, and it starts confidently making things up. The instinct is to blame the model and reach for a bigger one. That will not fix it, because the model is not where this breaks.
The model summarized the wrong thing
A retrieval-augmented system has two halves. First it retrieves: it searches your documents for passages relevant to the question. Then it generates: it hands those passages to the language model and asks it to answer using them. When the answer is wrong, people assume the second half failed. Usually the first half did.
If retrieval hands the model the wrong passages, the model does its job perfectly on bad material. It faithfully summarizes text that does not contain the answer, and a faithful summary of the wrong source is a confident, well-written, wrong answer. Swapping in a larger model makes the summary more fluent. It does not make the retrieved passages correct.
Where retrieval actually goes wrong
Four failure points cause most of the confident-wrong answers, and they are all in the retrieval half.
Chunking that breaks meaning. Documents get split into chunks before they are indexed. If the split lands badly, it can separate a table from the heading that explains it, or cut a rule off from its exception. The chunk that gets retrieved is technically about the right topic and missing the piece that makes it true.
Embeddings that match topic, not specifics. Retrieval usually works by semantic similarity, finding text that is about the same subject. But “our refund policy for annual plans” and “our refund policy for monthly plans” are about the same subject and say opposite things. Similarity search happily returns the one that sounds right and is wrong for this question.
No reranking. The initial search returns a list of candidate passages, and the best one is not always at the top. Without a second pass that reorders candidates by how well they actually answer the question, the model may be handed passages ranked eighth and ninth while the right one sits lower and unused.
No citations. When the system cannot show which source each claim came from, nobody can tell a grounded answer from an invented one. The failure is invisible, which means it goes unfixed.
Fix retrieval in order of impact
Work the problem where it lives, and do the high-leverage fixes first.
Start with chunking, because it is upstream of everything. Split documents in a way that respects their structure, keeping headings with their content and not slicing through tables or lists. Better chunks improve every query that follows.
Add reranking next. After the initial similarity search returns candidates, run a reranking step that scores each candidate on how well it answers the specific question and reorders them. This single addition often produces the largest jump in answer quality, because it fixes the common case where the right passage was retrieved but buried.
Then address specificity. Where your content has near-duplicate passages that differ in a critical detail, help retrieval tell them apart, through metadata, filtering, or clearer source structure, so a question about annual plans cannot be answered from the monthly-plan passage.
Citations are a correctness feature
Treat citations as essential, not decorative. When every claim in an answer links back to the source passage it came from, three things happen. Users can verify the answer instead of trusting it blindly. You can see, at a glance, when the model answered from nothing. And when retrieval fails, the failure is visible and therefore fixable.
Research on how language models use long contexts has shown they do not attend evenly to everything they are given: information in the middle of a long context is easy for the model to overlook. Citations surface exactly this kind of failure, showing you when the answer ignored the passage that actually held the truth.
Q: What is the refund window for annual plans?
A: Annual plans can be refunded within 30 days of purchase [1]. After 30 days, annual plans are non-refundable but can be cancelled to stop the next renewal [2]. Monthly plans follow a separate 14-day policy [3].
Sources
[1] Billing Policy v4, section 3.2 “Annual refunds”
[2] Billing Policy v4, section 3.4 “Cancellation”
[3] Billing Policy v4, section 3.1 “Monthly refunds”
Each claim links back to the exact passage it came from. When retrieval fails, a missing or wrong citation makes it visible instead of hiding inside a confident answer.
The takeaway
When your RAG system is confidently wrong, resist the urge to change the model. Instrument retrieval, add citations so you can see what it fed the model, and fix the chunking, ranking, and specificity problems that citations reveal. The model was never the weak link. The pipeline that feeds it was.
Building RAG systems that stay accurate on real questions, with retrieval you can trust and answers you can verify, is core to what NukyLabs does. If your bot demos well and fails in the wild, the fix is almost always in the half everyone ignores.
Facing this in your own build?
NukyLabs helps founders take AI-generated apps, agents, and automations from a working demo to something that survives real users. If any of the above hit close to home, we can scope it with you.
Get a free consultation →or message us to talk through your project.