RAG Is a System, Not a Prompt: Retrieval, Context, Evaluation

RAG answer quality is determined at four separate stages, each testable independently; treating it as a single system that is either "working" or "not working" makes it impossible to improve systematically.

The RAG Quality Gap: Why Prompt Engineering Isn't Saving Your Demo

Most Retrieval-Augmented Generation (RAG) projects begin with a moment of pure developer magic. You index a few PDFs in a vector store, wire up a basic similarity search, and pipe the results into an LLM with a template like "Answer based on the context." The first three questions you ask work perfectly. The demo looks like a production-ready product.

Then you hand it to a subject matter expert (SME). Within ten minutes, they find a failure. The LLM hallucinates an answer that isn't in the docs, or it claims it doesn't know the answer despite the information sitting right there in the corpus.

The standard engineering reaction to this is "prompt tuning." You spend three days wordsmithing the system instructions, begging the LLM to be more faithful, adding "take a deep breath," or formatting the context in XML blocks. Sometimes it helps. More often, it fixes one edge case while breaking three others. You are treating the RAG pipeline as a black box — a monolithic prompt where you put data in and pray for quality out.

This is the "prompt fallacy." In my experience, in an enterprise-grade system the prompt is the least significant variable in the quality equation. RAG is not a prompt; it is a distributed system with at least four distinct failure surfaces: Retrieval, Context Shaping, Generation, and Operations. If you aren't measuring these stages independently, you aren't engineering a system — you're just gambling with tokens.

Document Preparation — The Garbage In Gap

Before a single query is ever issued, your quality is already being constrained by your document ingestion pipeline. This is often the most neglected step of RAG — document preparation.

If your system relies on technical PDFs, your biggest enemy is not the LLM — it's the PDF format itself. Columns, headers, footers, and complex tables often turn into a "text lasagna" when using standard scrapers. If a table's numbers are flattened into a single list of digits, no retrieval algorithm can reconstruct the relationship between those numbers.

Document Quality Metrics

Parse Integrity: Does the extracted text maintain the semantic hierarchy (headings, bullet points)?
Table Reconstruction: Are tables preserved in a format (like Markdown or JSON) that an LLM can understand?
OCR Accuracy: In scanned documents, what is the character error rate? Even a 5% error rate can turn TX-900 into TK-800, making retrieval impossible.

Stage 1: Retrieval Quality — The Invisible Ceiling

Retrieval is the bedrock of RAG. If your retrieval engine fails to surface the correct information, no amount of prompt engineering or "frontier model" intelligence can save the response. Retrieval failures often masquerade as generation failures. If the LLM says "I don't know," is it because it followed your instructions to be cautious, or because the relevant chunk was 11th in a list where you only retrieved 10?

Vector Search vs. Hybrid Search

Traditional vector search (dense embeddings) is excellent at capturing semantic relationships — the "vibe" of a query. It knows that "How do I fix a leaky pipe?" is related to "plumbing repair." However, it is notoriously poor at handling specific identifiers, SKU numbers, or rare technical terms.

In enterprise environments, common queries often look like: "What is the torque spec for the TX-900 bolt?" A vector embedding might find three different bolt manuals, but not necessarily the TX-900, because "bolt" and "torque" dominate the semantic space.

This is where Hybrid Search becomes mandatory. By combining vector search with keyword-based search (like BM25), you ensure that exact matches are prioritized alongside semantic matches.

The Role of Reranking

Retrieval often retrieves a "candidate set" of, say, 50 documents using simple vector math. Passing 50 documents to an LLM is a recipe for noise and high costs. However, taking the top 5 by cosine similarity is risky because embeddings are "lossy" representations: cosine similarity measures general semantic relatedness, but it doesn't tell you whether a specific chunk actually answers the query (Microsoft Learn).

A Reranker (a Cross-Encoder model) performs a "second pass" over the candidate set. It looks at the query and the document simultaneously to determine their match strength — a cross-encoder takes the query and a candidate chunk as a single input pair, unlike a bi-encoder, which encodes them independently (Microsoft Learn). This is more expensive than vector math: in Microsoft's published benchmark, the semantic ranker reranked 50 documents of 2048 tokens in 158 ms on average, shorter documents took 50–113 ms, and the previous model version took up to 243 ms (Microsoft whitepaper). The order of magnitude depends on the model, document length, and hardware — a CPU-hosted cross-encoder easily runs into hundreds of milliseconds. It significantly improves the quality of the "ingredients" you send to the LLM.

Advanced Retrieval Pattern: HyDE

If your user queries are short or poorly phrased ("policy for travel?"), they may not have enough semantic density to find good chunks. One technique to fix this is HyDE (Hypothetical Document Embeddings), from Gao et al. (2023, arXiv:2212.10496). You ask an LLM to write a hypothetical document that answers the query — it may contain invented details. That text is then encoded by the same encoder your retriever uses, and the search runs on that vector. The rationale: the dense encoder acts as a lossy compressor, filtering the hallucinated details out and keeping the relevance pattern.

Stage 2: Context Quality — Signals in the Noise

Once you have the right chunks, you must decide how to present them. Even with the right data, the LLM's ability to reason is affected by the context's structure and length.

The "Lost in the Middle" Problem

In 2023, Liu and colleagues showed on a multi-document QA task that models use mid-context information notably worse: the accuracy curve is U-shaped — best at the beginning (primacy bias) and the end (recency bias). Their setup used 10–30 documents of 100 tokens each, and GPT-3.5-Turbo's accuracy dropped by more than 20%, in the worst case falling below the model with no documents at all (closed-book, 56.1%) — Lost in the Middle, arXiv:2307.03172.

To fight this, we use Context Shaping:

Relevance Sorting: Ensuring the reranker's top results are at the very top of the prompt.
Diversification: If you have 5 very similar chunks, don't include all of them. Include the most relevant one and pick others that cover different parts of the semantic space to avoid redundancy.

Noise Ratio and Context Coverage

Noise share (our working definition, not a RAGAS metric): if you retrieve 5 chunks but the answer is only in 1, 80% is noise. The closest RAGAS metric is Noise Sensitivity. A high noise share makes the model latch onto "distracting passages" — in Cuconasu et al. (2024, arXiv:2401.14887)'s terminology, topically relevant chunks that do not contain the answer, as opposed to random irrelevant ones, which barely hurt answer quality.
Context Recall: If the answer requires facts A, B, and C, and your retriever only found A and B, the system will fail or hallucinate C. Measuring context recall ensures you have the "coverage" needed for a complete answer.

Stage 3: Answer Quality — Grounding and Faithfulness

At the generation stage, we are evaluating the LLM's adherence to the context.

The "Attribution Error"

A common RAG failure is when the LLM uses its training data to "fill in the gaps" of the provided context. If your manual says "Product A is blue" but the LLM's general knowledge says "Product A is often red," it may get it wrong. This is an attribution error.

We measure this using Faithfulness, from the RAGAS metric set (Es et al., 2023, arXiv:2309.15217):

Extract every claim made in the LLM's answer.
Check each claim against the retrieved context.
Faithfulness = (Claims supported by context) / (Total claims) — this is how RAGAS defines the metric.

Answer Relevance vs. Context Recall

Your answer could be 100% faithful (contains no lies) but still irrelevant if it didn't actually answer what the user asked. Conversely, a relevant answer might have low context recall if the LLM ignored your "only answer based on the context" rule and just used its pre-existing knowledge. You need both metrics — Response Relevancy (renamed from Answer Relevance in RAGAS) and Context Recall — to tell the full story.

Stage 4: Operational Quality — The Price of Production

A RAG system that is 99% accurate but takes 45 seconds to answer is a failure for most enterprise use cases. Operational quality is where your architecture meets user patience and the company budget.

P99 Latency vs. Average Latency

In RAG, your latency "long tail" is brutal. If one document retrieval takes longer, or the LLM generates a longer-than-usual explanation, the user waits.

Time-to-First-Token (TTFT): This is what makes a UI feel "snappy."
Generation speed (tokens per second): How fast the answer streams once started.
Reranker Overhead: If your reranker adds 1 s to every query, is the quality gain worth the friction?

Semantic Caching

To reduce costs, you can implement a semantic cache. Before running a retrieval/LLM loop, check if a "semantically identical" query has been answered recently. If a user asks "What is the Wi-Fi password?" and an hour ago someone asked "how to join the guest network?", you can likely serve the same answer. This saves tokens and time, but it carries the risk of serving stale information once the document corpus has been updated.

The Quality Model Artifact: A Four-Stage Framework

Stage	What it Measures	Metric Examples	What it Tells You	What it Misses
Retrieval	Searching for the right data.	Recall@k, MRR, Hit Rate.	Do we have the raw ingredients for the answer?	If the information is formatted in a way the LLM can use.
Context	Preparing the data for the LLM.	Context Precision, Noise Sensitivity.	Are we overwhelming the model with noise or redundant data?	If the LLM is smart enough to generate a correct response.
Answer	Factual accuracy and grounding.	Faithfulness and its inverse — the share of unsupported claims.	Is the LLM being honest and helpful relative to the context?	Why the answer is wrong (retrieval vs. logic).
Operations	Real-world reliability and cost.	P99 Latency, Cost/Query, TTFT.	Is the system viable for production users and scale?	The actual correctness of the answers.

Strategy: Automated Metrics vs. Human Review

When to trust an automated metric (RAGAS / LLM-as-judge)

For Regression Testing: Use RAGAS in your CI/CD pipeline. If a code change drops your Faithfulness score across a set of 100 test queries, stop the deployment.
For Hyperparameter Tuning: Use automated metrics to decide if top_k=5 is better than top_k=7. Humans cannot look at enough data to make this statistically valid.

When Human Review is necessary

Golden Set Initialization: An expert must define the first 50–100 queries and their "ideal" source documents. This is my practical recommendation, not an industry norm. This is the only way to "ground" your automated metrics.
Logic-Heavy Tasks: Automated judges often fail to spot complex missing logic (e.g., "The model forgot to adjust for the tax rate mentioned in Section 4").

10 Common RAG Failure Modes (And How to Spot Them)

The Silent Miss: Retrieval found nothing, but you have no "out of scope" detector, so the LLM guesses. (Stage 1)
Token Overload: You sent 15,000 tokens of context and the LLM "forgot" the facts in the middle. (Stage 2)
The "Stale Chunk": Retrieval found an old version of a document. (Document Preparation)
Semantic Drift: Your embedding model wasn't trained on your technical jargon, so it thinks "Apple" (the fruit) and "Apple" (the company) are the same thing. (Stage 1)
Formatting Sabotage: You passed a poorly parsed table that looks like a wall of text. (Document Preparation)
Prompt Instruction Conflict: You told the LLM to "be concise," so it cut out half the required facts. (Stage 3)
Latency Death Spiral: You added a reranker and HyDE, and now every query takes 12 seconds. (Stage 4)
The False Positive: RAGAS says the answer is 100% faithful because the LLM cited the docs correctly, but the docs themselves are wrong. (Document Preparation)
Budget Bleed: You are indexing 1M documents that no one ever queries, paying for massive vector store clusters. (Stage 4)
The Citation Lie: The LLM provides a correct answer but points to the wrong document as the source. (Stage 3)

Conclusion: Engineering the System

If you take only one thing from this article, let it be this: stop changing your prompt until you have measured your retrieval recall.

RAG is not a magic solution to information search; it is an assembly line. Every stage of that line — Retrieval, Context Shaping, Generation, and Operations — can fail, and document preparation sets the ceiling for everything downstream. Treating the whole thing as a single black box is a recipe for frustration and unreliable production systems.

By measuring each stage independently, you turn a "vibe-based" AI project into a predictable engineering project. You can debug failures, you can set performance SLAs, and most importantly, you can sleep at night knowing your system isn't just a very expensive hallucination engine.

Implementation Checklist for Seniors

Retrieval: Implement a "Golden Set" (50+ queries) and measure your Recall@k. If it's below 80% — my working threshold, not an industry norm — stop prompt tuning and start hybrid search / reranking.
Context: Monitor your noise share. If you're sending 10 chunks to get 1 answer, your context is too noisy.
Generation: Run an automated "Faithfulness" check on a 10% sample of production queries — a sensible starting point, not a figure backed by public sources. Watch for when the LLM brings in outside knowledge.
Operations: Track P99 latency and Cost-per-Query. Use a semantic cache if you have high volume on repetitive queries.

Sources

Related Articles:

Open Source Projects:

Agent Shaker — task-oriented multi-agent orchestration for .NET.

Discuss an enterprise AI architecture or delivery challenge → techbuzzz.ru