RAG & Grounding

⏱ ~9 min read · 🎧 10 min listen · ✎ 3 quizzes · 🧪 ~90 min lab

Every FDE posting on your gap map asks for RAG — Databricks names it explicitly, right next to multi-agent systems and evals.⁶ But here's the field reality behind the buzzword: most RAG systems that fail in the enterprise don't fail because someone picked the wrong vector database. They fail because retrieval returns the wrong chunks, or because the model answers confidently even when retrieval returned nothing useful. In your capstone domain — a virtual-care provider whose clinicians will ask an assistant about clinical protocols — a confidently wrong answer isn't an embarrassing demo moment. It's a safety incident. This lesson is about engineering the two things that actually decide whether a RAG system is trustworthy: retrieval quality and grounding.

What RAG actually is — and where it actually breaks

RAG is one idea: instead of hoping the model memorised your customer's documents, you retrieve the relevant passages at question time and put them in the context window, then ask the model to answer from those passages. Everything else — embeddings, vector stores, rerankers — is plumbing in service of that idea. The practitioners' consensus from a year of production deployments is blunt: teams over-invest in the database and under-invest in retrieval quality, and hybrid keyword-plus-semantic search plus sensible chunking beats an exotic vector store almost every time.³

When a RAG system gives a wrong answer, an FDE's first job is diagnosis, and there are exactly two places to look. A retrieval failure means the right passage never made it into the context — the model literally couldn't have answered correctly. A grounding failure means the right passage was in the context and the model ignored it, blended it with its own prior knowledge, or answered a question the documents don't cover. The fixes are completely different, which is why guessing is expensive:

Retrieval quality I: chunking is a real decision

Documents get split into chunks before embedding, and how you split them is a genuine engineering trade-off, not a default to accept. Small chunks embed precisely — the vector captures one idea — but strip away the context that makes the idea interpretable. Large chunks keep context but blur the embedding, so retrieval gets noisier and you burn context-window tokens on padding. And naive fixed-size splitting can cut a protocol's dosage table away from the heading that says which patients it applies to. The durable advice: chunk along the document's own structure (sections, headings, paragraphs) before reaching for fancier schemes, and test retrieval on real questions before trusting any strategy.³

Retrieval quality II: contextual retrieval. Anthropic's technique attacks chunking's core problem — a chunk that says "the escalation threshold is 39.5°C" loses meaning once separated from the document titled "Paediatric fever protocol". The fix: before embedding each chunk, have a model prepend a short chunk-specific explanation situating it in its source document. Combined with BM25 keyword matching, this cut retrieval failures by 49%; adding reranking — a second model that re-scores the top candidates for relevance before they reach the context — pushed the reduction to 67%.¹ That's a two-thirds cut in the dominant failure mode, from preprocessing and a reranker, with no exotic infrastructure. You'll implement the contextual half in this lesson's lab and measure the improvement yourself.

Now grounding — the other half of trust. Anthropic's guidance for reducing hallucinations is refreshingly unmagical: explicitly allow the model to say "I don't know", and for document-based tasks, make it extract verbatim quotes first and cite them in the answer.² Both work for the same reason: they change the model's job from "produce a plausible answer" to "produce an answer supported by this evidence, or decline." For the virtual-care provider, that's the difference between an assistant that says "per §3 of the fever protocol, escalate at 39.5°C" and one that improvises a threshold. A citation the clinician can check in one click is a trust feature, an audit feature, and a debugging feature all at once.

When RAG is the wrong tool

This is the AI-judgment rep for this lesson: customers now ask for "a RAG chatbot" by name, and part of your job is knowing when to talk them out of it. If the whole corpus is small — say, under a few hundred thousand tokens — you may not need retrieval at all: current models take 1M-token context windows, and putting the documents directly in context sidesteps every retrieval failure mode at the cost of more tokens per call.⁵ If the question needs live data — "what's this patient's current medication list?" — the answer is a tool call to the system of record (Lesson 03), not a stale document index. And if the customer wants deterministic answers to a small fixed set of questions, a curated FAQ beats any model. Persuasive models make bad architectures look good in demos; the FDE is paid to notice.⁷

🧪 Practical steps: grounded policy Q&A with before/after numbers (~90 min)

This lab produces the portfolio artifact hiring managers ask about in RAG interviews: not "I built a chatbot" but "I measured retrieval failures, applied contextual retrieval, and cut them — here are the numbers." You'll build a grounded Q&A system over synthetic clinical protocols for the virtual-care provider, with mandatory citations and honest refusals. Cost and latency tuning waits for Lesson 09 — today is about correctness. Full instructions live in labs/0004-grounded-policy-qa.md; the shape:

1. Generate a synthetic corpus — use Claude to write ~10 short clinical protocol documents (triage escalation, telehealth prescribing, after-hours workflows…) for the fictional provider. Generating realistic synthetic data is itself an FDE skill — you'll lean on it hard in the Lesson 11 capstone.
2. Build naive RAG — fixed-size chunking, embed or keyword-score the chunks, retrieve top-k into the prompt.
3. Enforce grounding — the answer prompt requires a verbatim quote plus a citation (doc + section) for every claim, and instructs the model to answer INSUFFICIENT_CONTEXT when the retrieved passages don't support an answer.²
4. Write 10 test questions with expected source sections — including 2 deliberately unanswerable ones (topics your corpus doesn't cover). Those two are where most RAG demos secretly fail.
5. Measure — for each question, log whether the correct chunk was retrieved (retrieval), whether the answer cited it faithfully (grounding), and whether the unanswerable questions were refused. Count failures by type.
6. Add contextual retrieval — use Claude to prepend a situating sentence to every chunk (the technique from the primary source¹), re-index, re-run.
7. Report before/after — a small table in README.md: retrieval failures, grounding failures, unanswerables handled, naive vs contextual. Honest numbers only — "contextual retrieval fixed 2 of my 3 retrieval failures" is a better interview story than fake perfection.
8. Save everything to D:\Projects\FDE-Portfolio\l04-rag-qa\.

Feedback loop: bring your before/after table and one failing question to me in chat and I'll review it like a customer's sceptical clinical lead — are the numbers honest, is every answer actually traceable to a quote, and did the unanswerable questions get refused rather than improvised? This artifact feeds the Lesson 11 capstone engagement and is tracked in the course's artifact tracker.

You are a senior Forward Deployed Engineer reviewing my work: a grounded RAG Q&A system over ~10 synthetic clinical-protocol documents for a virtual-care provider. It includes naive-chunking RAG, mandatory quote+citation grounding with refusal on unsupported questions, a 10-question test set (2 deliberately unanswerable), and a before/after measurement comparing naive chunking vs contextual retrieval (chunk-context prepending).

Grade each criterion as Strong / Adequate / Missing, with one sentence of evidence:
- Measurement honesty: failures are counted per type (retrieval vs grounding), the before/after numbers are plausible for a 10-question set, and remaining failures are reported rather than hidden.
- Citation enforcement: every answered question carries a verbatim quote plus a doc+section citation that actually appears in the corpus — spot-check two.
- Handling of unanswerables: both out-of-corpus questions produce a refusal (INSUFFICIENT_CONTEXT or equivalent), not a plausible-sounding improvised answer.

Be skeptical — pick the answer most likely to be subtly ungrounded and trace its citation back to the source text. Then ask me 2-3 follow-up questions a clinical-safety reviewer would ask. Finish with the single highest-leverage improvement before this could face real clinicians.

My report and code follow below.

Check yourself — think like an FDE

Scenarios, not recall. Diagnose from the mental model — don't scroll up. Wrong picks stay live.