Cost, Latency & Eval-Driven Iteration

⏱ ~10 min read · 🎧 11 min listen · ✎ 3 quizzes · 🧪 ~90 min lab

Here's a failure mode that quality metrics never catch: the pilot works, users like it, the evals are green — and then finance runs the numbers. Twelve dollars per query, eleven seconds per answer, and the business case that justified the engagement quietly dies. Remember the number this course opened with: roughly 95% of enterprise GenAI pilots show no measurable impact⁸ — and a system whose unit economics don't work at production volume is one of the quietest ways to join that statistic. Lesson 08 made your system trustworthy. This lesson makes it affordable and fast — without breaking what you've built.

The bill arrives in week three

In a demo, nobody asks what a query costs. In production, someone multiplies your per-query cost by the customer's real volume — and that multiplication decides whether the engagement survives. The numbers get big fast: Anthropic's own multi-agent research system uses about 15× more tokens than a chat interaction, which is only economically viable for tasks whose value justifies it.⁷ An FDE who can't state their system's cost per query and p95 latency from memory hasn't finished hardening it. So before optimizing anything, instrument: tokens in, tokens out, cache hits, cost per query, and latency at the 50th and 95th percentile. You cannot cut what you haven't measured.

Eval-driven iteration: the discipline that makes speed safe

Every optimization in this lesson changes model behavior or model inputs — which means every one of them can silently break quality. The discipline that makes fast field iteration safe is simple: the eval suite from Lesson 05 runs before and after every change, and a change only ships if the evals stay green.⁶ That's what "eval-driven iteration" means: your evals aren't a one-time proof the system works — they're the regression gate that lets you swap models, prune context, and tune prompts at field speed without fear. Optimize without that gate and you're not iterating, you're gambling with the customer's trust.

Prompt caching: the biggest single lever

Prompt caching is usually the largest cost cut available for the least work. The idea: your system prompt, tool definitions, and document corpus are identical on every request — so instead of reprocessing them each time, the API caches the prefix. Cache reads cost 0.1× the base input price; writes cost 1.25× (5-minute TTL) or 2× (1-hour TTL), so a 5-minute cache pays for itself after a single reuse.¹ The simplest path is automatic caching — one top-level cache_control field and the API manages breakpoints as the conversation grows.¹ Two field-relevant details: caching is a prefix match, so a timestamp interpolated into your system prompt silently kills every cache hit downstream; and since February 2026, caches are isolated per workspace rather than per organization¹ — if your customer's deployment spans workspaces, each one warms its own cache. Always verify with usage.cache_read_input_tokens in the response: if it's zero on repeated requests, something is invalidating your prefix.

Model-tier routing: big model where judgment matters

Not every query needs your most capable model. The routing principle: use the big model where judgment matters, the small model where it doesn't — and prove "it doesn't" with evals, not vibes. As of mid-2026 the tier spread is wide: Claude Opus 4.8 costs 5× more than Claude Haiku 4.5 on both input and output tokens,² so routing even a third of traffic down a tier is a serious saving. The eval suite is what makes model routing honest: run your test set against the cheaper model, per query category. Where the small model passes, route to it. Where it fails, it has just proven it needs the big model — and you now have evidence to show the customer either way. Routing on intuition alone is the classic way an optimization pass turns into a quality incident nobody notices until renewal.

Latency budgets: fix the right kind of slow

Cost has one currency; latency has several, and customers feel them differently. A latency budget splits your target across stages — retrieval, time-to-first-token, generation — so you fix the stage that's actually slow. OpenAI's latency guide distills this into seven principles: process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelize, make your users wait less, and don't default to an LLM at all.³ Two of those do most of the field work. Generate fewer tokens — output tokens are both the most expensive and the slowest, so a one-line "answer concisely" instruction is often the cheapest latency win available. And make your users wait less — streaming doesn't change total latency, but showing the first token in one second instead of a blank screen for nine transforms perceived speed. Note the last principle is the AI-judgment thread again: sometimes the fastest LLM call is the one you replace with a lookup table.

Token diets: prune what the model reads

The remaining lever is shrinking what goes in. Retrieval is the usual offender in a RAG system: if you're stuffing ten chunks into every request and the answer is in the top three, seven chunks are pure cost — and long, noisy context degrades quality too, the "context rot" problem from Lesson 02.⁴ For long-running conversations and agents, the API offers compaction — summarizing earlier context server-side — and context editing, which clears stale tool results entirely.⁴ Tool-heavy agents have a further trick: loading tool definitions on demand and executing tool calls as code instead of round-tripping every intermediate result through the model, which Anthropic reports cutting an example agent's token bill from ~150K to ~2K tokens.⁵ Good context engineering and good cost engineering turn out to be the same discipline.

🧪 Practical steps: the cost/latency pass (~90 min)

You're going to do to your Lesson 04 RAG system exactly what you'd do in week three of a real engagement: instrument it, cut its cost by at least 40%, and prove quality didn't move. The artifact is a before/after optimization report plus the instrumented code — a portfolio artifact that answers the interview question "tell me about a time you optimized an LLM system" with real numbers. Full instructions live in labs/0009-cost-latency-pass.md; save everything to D:\Projects\FDE-Portfolio\l09-optimization\.

1. Copy your L04 RAG Q&A into l09-optimization\ so the baseline stays untouched.
2. Instrument it: wrap every API call to record input/output/cache tokens from response.usage, cost per query (from the pricing table above), and wall-clock latency. Run your 10 test questions 3× and write baseline.json with totals, cost per query, and p50/p95 latency.
3. Add prompt caching: put cache_control on the system prompt and corpus context. Verify cache_read_input_tokens > 0 on the second request — if it's zero, hunt the invalidator.
4. Add model-tier routing: classify each query as easy or hard (a keyword heuristic or a cheap claude-haiku-4-5 classifier call) and route easy ones to claude-haiku-4-5, keeping hard ones on your L04 model.
5. Prune retrieved context: reduce top-k and/or cap context tokens. Find the smallest retrieval that still passes the evals.
6. Run the regression gate: your full L04 test set plus your L05-style checks, after each optimization — not just at the end. If a change turns an eval red, revise or revert it.
7. Write report.md: a before/after table (cost per query, total tokens, cache hit rate, p50/p95 latency, eval pass rate), which optimizations you applied and why, and one you considered but rejected. Target: ≥40% cost reduction with evals green.

Feedback loop: bring your report back to me in chat and I'll review it like the customer's platform lead would — is the measurement methodology sound, does every optimization carry a justification, and can you prove quality held? This artifact feeds the Lesson 11 capstone engagement and is tracked in the course's artifact tracker.

You are a customer's skeptical platform engineering lead reviewing my work: a cost/latency optimization pass on a RAG Q&A system built on the Claude API. I instrumented tokens, cost per query, and p50/p95 latency; then applied prompt caching, model-tier routing, and retrieved-context pruning; and re-ran my evaluation suite after each change. My deliverable is a before/after report.

Grade each criterion as Strong / Adequate / Missing, with one sentence of evidence:
- Measurement rigor: the baseline is measured (multiple runs, real usage fields from the API, p50 AND p95), not estimated — and before/after numbers are comparable.
- Optimization justification: each change is tied to a measured cost or latency driver, not applied blindly; at least one considered optimization was rejected with a reason.
- Regression proof: there is concrete evidence the eval suite stayed green after every change — including which model handled which queries under routing.

Be skeptical — attack the weakest number first (a suspiciously round figure, a single-run measurement, a missing cache-hit rate). Then ask me 2–3 follow-up questions a cost-conscious customer would ask before approving this for production. Finish with the single highest-leverage next optimization I haven't tried.

My optimization report follows below.

Check yourself — think like an FDE

Scenarios, not recall. Diagnose from the mental model — don't scroll up. Wrong picks stay live.