Trusted technology partner for 50+ companies worldwide

Retrieval-Augmented Generation

RAG Done Right —
What Nobody Tells You
About Retrieval in Production

Most RAG implementations work fine on your test data. They fall apart on real user questions. The gap is in retrieval quality — not the LLM.

💬

Query

User question

✂️

Chunk

Split & index

🔢

Embed

Vectorize

🔍

ANN lookup

🏆

Rerank

Cross-encode

🤖

LLM

Generate

Failure Modes

RAG Quality Levels

Thing That Beats a Bigger Model

The Hidden Problem

5 Ways RAG Fails Silently

Your RAG system returns answers — that's the dangerous part. Silent failures produce plausible-sounding responses backed by the wrong context. Here are the five culprits, and what they look like in practice.

Failure Mode 01

Wrong Chunk Size

Too small: chunks lack the context the LLM needs. Too large: the relevant signal is buried in noise. There is no universal default — optimal size depends on your content structure and query patterns.

Too small

…

Optimal

context

Too large

noise · noise · signal · noise · noise

Failure Mode 02

Wrong Embedding Model for the Domain

A model trained on Wikipedia won't understand "T-cell exhaustion" or "EBITDA margin compression." Domain-specific terminology collapses into generic vectors — semantic similarity becomes meaningless.

# Generic model — cosine distance
embed("myocardial infarction") ≈
embed("heart attack")  dist=0.82 ✗
# Domain-tuned model
embed("myocardial infarction") ≈
embed("heart attack")  dist=0.97 ✓

Failure Mode 03

No Reranking Step

ANN retrieval optimizes for speed, not relevance. Cosine similarity in embedding space is a proxy metric. A cross-encoder reranker evaluates query-document pairs directly — often flipping the top result entirely.

# Without reranking (top-3 by cosine)
rank 1 score=0.91 ← partially related
rank 2 score=0.88 ← off-topic
rank 3 score=0.85 ← actual answer
# After cross-encoder reranking
rank 1 score=0.97 ← actual answer ✓

Failure Mode 04

Missing Metadata Filters

Pure semantic search has no concept of time, version, or source type. Without filters, your RAG happily retrieves a 2019 deprecation notice when the user asks about the current API.

# Without metadata filter
query: "how to authenticate"
↳ retrieved: docs/auth_v1.md (2019) ✗
# With metadata filter
query: "how to authenticate"
filter: {"version": "≥3.0", "type": "guide"}
↳ retrieved: docs/auth_v3.md (2024) ✓

Failure Mode 05

No Retrieval Evaluation

Teams measure LLM output quality, but never measure whether the right chunks were retrieved in the first place. You optimize the wrong layer and wonder why accuracy doesn't improve.

The Path Forward

The RAG Quality Ladder

Five levels separate a weekend prototype from a production system you can trust. Each level builds on the last — you can't skip rungs without accumulating hidden debt.

Level 1 — Starting Point

Naive RAG — Basic Vector Search, No Eval

Fixed-size chunks, one embedding model, top-k cosine search, feed directly to LLM. Works on demos. Fails on edge cases you haven't seen yet. You have no idea how often it fails because you have no measurement.

text-embedding-ada-002chunk_size=512top_k=5no eval ✗

Level 2 — Smarter Indexing

Chunking Strategy + Metadata Enrichment

Semantic chunking respects document structure. Metadata tags (date, source, version, section) enable pre-filtering before vector search. Parent-document retrieval fetches larger context windows around matched chunks.

semantic chunksmetadata filtersparent-doc retrievalsliding window

Level 3 — Better Retrieval

Reranking + Hybrid Search (BM25 + Dense)

Combine keyword (BM25/TF-IDF) and dense vector retrieval — keywords catch exact matches that semantics miss. A cross-encoder reranker re-scores the combined candidate set. Reciprocal Rank Fusion merges the result lists.

BM25 + dense fusioncross-encoder rerankRRF mergingquery expansion

Level 4 — Measurement

Retrieval Eval Harness — MRR, NDCG, Hit Rate

A golden Q&A dataset with ground-truth relevant chunks. Automated metrics run on every retrieval change. You now know your Hit@3, MRR@10, and NDCG@5 — and regressions block deployment.

golden datasetMRR@10NDCG@5Hit@3CI gate

Level 5 — Production

Continuous Monitoring + Feedback Loop

Live retrieval metrics tracked per-query. Low-confidence retrievals are flagged. User feedback (thumbs down, corrections) flows back into the eval dataset. Model drift and distribution shift are caught automatically. The system gets better with every failure.

live metricsfeedback ingestiondrift detectionauto-reindexingA/B retrieval tests

The Root Cause

Hallucination vs. Retrieval Failure

Most "LLM hallucinations" in RAG systems are actually retrieval failures in disguise. The LLM is doing its job — it's faithfully working with the wrong context. Getting the diagnosis right determines where you fix it.

Retrieval Failure

Wrong chunks retrieved — LLM has nothing to work with

The retrieved context doesn't contain the answer. The LLM either says "I don't know" (good) or — more often — confabulates a plausible answer from its parametric knowledge (bad). This looks like a hallucination but is actually a retrieval miss.

1Log retrieved chunks — is the answer actually in them?

2Check retrieval metrics: Hit@3, MRR — are the right docs ranked?

3Fix: improve chunking, add reranker, tune embedding model

4Verify with a prompt that gives the correct chunk directly

True LLM Failure

Correct chunks retrieved — LLM ignores or distorts them

The retrieved context does contain the answer, but the LLM fails to extract it correctly. This is a genuine generation failure — prompt design, context window position bias, or instruction-following issues.

1Confirm the correct chunk is in retrieved context

2Measure faithfulness score against retrieved context

3Fix: improve system prompt, chunk ordering, context framing

4Consider a stronger/larger LLM or fine-tuned generation layer

Key Diagnostic Question

Before blaming the LLM: inspect the retrieved chunks. If you paste the correct context directly into the prompt and the LLM answers correctly, you have a retrieval problem. If it still fails — you have a generation problem. These require completely different fixes. Debug in order: retrieval first, generation second.

Measure Everything

What a Production RAG Eval Looks Like

A golden Q&A dataset drives retrieval metrics. Generation metrics measure faithfulness independently. A combined eval pipeline runs in CI and gates every change.

The Eval Stack

Layer 1 — Golden Dataset

~200 hand-curated query / relevant-chunk / expected-answer triples. Cover head queries (80%) and tail edge cases (20%). Update as content changes.

Layer 2 — Retrieval Metrics

For each query, check if the relevant chunk appears in top-k results. Compute Hit@3, MRR@10, NDCG@5. Alert if MRR drops more than 5% from baseline.

Layer 3 — Generation Metrics

Faithfulness (does the answer contradict the retrieved context?), Answer Relevance (does it answer the question?), Correctness vs. ground truth. Use an LLM-as-judge or RAGAS framework.

Layer 4 — CI Gate

Every pull request that changes chunking, embedding, or retrieval logic runs the full eval suite. Regressions block merge. Improvements get logged.

rag_eval_report.py — run complete

## ─── RAG Eval Pipeline v2.3 ───
## Dataset: golden_queries_v4.json (214 items)
## Retriever: hybrid-bm25+dense + cross-encoder
RETRIEVAL METRICS ─────────
MRR@10
0.71▲ +0.08 vs baseline
NDCG@5
0.76▲ +0.11 vs baseline
Hit@3
0.84▲ +0.06 vs baseline
Hit@1
0.62▲ +0.09 vs baseline
GENERATION METRICS ─────────
Faithfulness
0.91✓ above threshold
Answer Relevance
0.88✓ above threshold
Correctness
0.79✓ above threshold
Context Recall
0.83✓ above threshold
─────────────────────────────
Failures breakdown:
  retrieval_miss   : 31 (14.5%)  ← fix retrieval
  generation_error :  8 ( 3.7%)  ← fix prompt
  correct          : 175 (81.8%)
✓ CI gate: PASSED — safe to merge

The Bottom Line

Better retrieval beats a bigger model every time.

You don't need GPT-5. You need the right chunks in the context window. Measure your retrieval. Fix what's broken. Ship with confidence.

Chunk Smarter·Embed Better·Rerank Always·Filter by Metadata·Measure Retrieval

Build RAG That Actually Works

Retrieval-first AI, measured and monitored — not vibes.

RAG Done Right —What Nobody Tells YouAbout Retrieval in Production