Evals That Prove It Works

⏱ ~10 min read · 🎧 9 min listen · ✎ 3 quizzes · 🧪 ~120 min lab

Here's the moment this lesson is built for: your demo landed, the customer is excited, and someone asks, "how do we know it'll still work next month?" A demo shows the system worked once, on inputs you picked, on a good day. An eval proves it keeps working — on inputs you didn't pick, after every prompt tweak, model upgrade, and data change. The market is explicit about this: Anthropic's FDE posting requires production LLM experience including "evaluation frameworks"⁴, and Databricks demands "expertise in deploying production-grade GenAI applications, including evaluation and optimizations."⁵ Evals aren't a QA afterthought. They're a named hiring bar.

The deliverable nobody asks for — until it's missing

Hamel Husain's field observation, from consulting across dozens of AI teams: unsuccessful AI products almost always share one root cause — a failure to create robust evaluation systems.¹ The demo-to-production gap you met in Lesson 01 is, mechanically, an eval gap. Without evals, every change to the system is a gamble: you improve one case and silently break three others, and nobody notices until the customer does. For an FDE this has a sharper edge — the eval harness is often the artifact that survives you. You leave the site in six weeks; the harness is how the customer's own team keeps trusting, and safely changing, what you built.

Evals start before code, with success criteria. Anthropic's guidance: make criteria specific, measurable, and tied to the use case — not "the summaries should be good" but "the summarizer achieves an F1 score of at least 0.85 on the held-out set, and never includes patient-identifying information."³ The FDE move is to extract these numbers from the customer in discovery: what would make you trust this? What failure would make you switch it off? Those two answers become your first eval criteria.

The three-level framework

Hamel Husain's framework organises evaluation into three levels, distinguished by who or what does the judging and how often it runs.¹ You don't graduate from one level to the next — a healthy system runs all three at different frequencies:

Choosing a grader

Every eval case is a pairing of an input with a grader. Anthropic's test-development guide names the main grader families, and the skill is matching the grader to the claim you're trying to prove³ — exact match for claims with one right answer, similarity scoring for claims about meaning, and an LLM judge for claims about quality that resist mechanical definition:

Never ship an unvalidated judge

The LLM judge is the most powerful grader and the most dangerous, because an unvalidated judge just automates your blind spots — you've replaced "the model seems fine" with "the judge says the model is fine", and proven nothing. Hamel's fix is critique shadowing: a domain expert writes pass/fail judgments with critiques on real examples, you iterate the judge's prompt until its judgments agree with the expert's, and you keep measuring that agreement over time.² The judge earns trust the same way a new team member does — by being checked against someone who knows. And critically, the expert must be the domain expert (the nurse, the claims assessor), not you.

Where do eval cases come from? The best ones are harvested from real failures. Every bug report, every "the AI got this wrong" complaint, every weird trace you notice in the logs becomes a permanent test case — this is how your eval set grows teeth over time. Hamel calls the habit "looking at your data": most teams skip it, and it's the single highest-leverage eval activity there is.¹ Until real users arrive, seed the set with synthetic cases — enumerate the dimensions of your inputs (note type, specialty, messiness) and generate combinations with an LLM.¹

🧪 Practical steps: an eval harness for your summarizer (~120 min)

In Lesson 02 you built a clinical-note summarizer and checked it by hand against five criteria. Time to industrialise those hand checks into a harness that catches regressions automatically — the single most quoted artifact type in FDE job postings. Full instructions live in labs/0005-eval-harness.md; you can hand-roll it in Python (shown, recommended first time) or use promptfoo⁷ — the lab supports both.

1. Curate an eval set — take your Lesson 02 test notes and grow the set to 10–12 cases (synthetic generation is fine), each saved as JSON with the input note and the facts that must and must not appear. This file is the portable asset.
2. Turn your 5 hand checks into Level-1 assertions — plain Python functions: output parses, required fields present, no invented medications, no patient identifiers, length within bounds.
3. Add an LLM judge with a written rubric — a second API call that grades each summary pass/fail on faithfulness and completeness, returning structured JSON with a critique.
4. Validate the judge — hand-grade 5 outputs yourself before looking at the judge's opinion, then measure agreement. Below 4/5, fix the rubric and rerun — that's critique shadowing in miniature.
5. Plant a regression — deliberately degrade the summarizer prompt (delete the grounding instructions from Lesson 04, or the "say only what's in the note" rule) and rerun.
6. Produce the before/after report — a markdown table of scores per case for the good and degraded prompts. The harness must visibly catch the plant.
7. Save everything to D:\Projects\FDE-Portfolio\l05-eval-harness\ — eval set, harness code, judge rubric, validation notes, and the report.

Feedback loop: bring your before/after report and your judge-validation notes back to me in chat, and I'll review them like a customer's tech lead would — do the assertions test real failure modes or just formatting, was the judge validated against human judgment honestly, and does the report prove the harness catches regressions? This artifact feeds the Lesson 11 capstone engagement and is tracked in the course's artifact tracker.

You are a senior Forward Deployed Engineer reviewing my work: an eval harness for a clinical-note summarizer. It contains an eval set of ~10 cases, coded assertions, an LLM judge with a written rubric, notes validating the judge against 5 of my own human judgments, and a before/after report showing the harness catching a deliberately planted prompt regression.

Grade each criterion as Strong / Adequate / Missing, with one sentence of evidence:
- Assertion quality: the coded assertions test real clinical failure modes (invented medications, leaked identifiers, missing critical facts), not just output formatting.
- Judge-validation rigor: I graded outputs myself BEFORE seeing the judge's opinion, measured agreement explicitly, and iterated the rubric when agreement was low — not just "the judge looks right".
- Regression-catch demonstration: the planted degradation is realistic, and the before/after report shows the harness flagging it clearly, with scores per case rather than a single vague number.

Be skeptical — probe whether my judge could be agreeing with me for the wrong reasons, and whether any assertion would pass on garbage output. Then ask me 2–3 questions a customer's tech lead would ask before wiring this harness into their CI. Finish with the single highest-leverage improvement.

My eval harness materials follow below.

Check yourself — think like an FDE

Scenarios, not recall. Diagnose from the mental model — don't scroll up. Wrong picks stay live.