All insights
AI Evaluation Engineering

Vibes Are Not
an Eval

Every team says their AI "works great" — until it doesn't. The difference between teams that catch regressions and teams that get surprised by them is a real eval harness. Vibes don't scale.

10k+Edge Cases
4Eval Layers
1Flywheel
eval-harness · regression-suite · v2.8.1
$ eval-suite run --suite regression --dataset golden-v4
Loading 10,247 test cases...
✓ [001/250] entity_extraction .......... PASS (94.2%)
✓ [002/250] rag_faithfulness ........... PASS (0.91)
✓ [004/250] agent_task_completion ...... PASS (87.3%)
⚠ [005/250] context_recall ............ WARN (0.74 < 0.80)
✗ [007/250] hallucination_rate ........ FAIL (8.3% > 5.0%)
... running 243 more cases ...
✓ [250/250] summarization_quality ...... PASS (0.89)
─────────────────────────────────────────
247/250 passed ✓ 3 regressions ✗
⚠ Deploy blocked — fix regressions first
progress100%
The Problem

Why "Manual Testing" Breaks Down

Clicking around in the playground is not quality assurance. Here's exactly where the wheels fall off — and why the gap between what you tested and what users experience widens every sprint.

01
You Can't Test 10,000 Edge Cases by Hand
Real user distributions contain rare but critical inputs: edge-case dates, ambiguous pronouns, adversarial phrasing. A human can cover maybe 50 before shipping. An eval suite covers 50,000 in under 3 minutes.
Manual~50Eval50k+⚡ 3 minvs hoursof clicking
02
Prompts That “Feel Better” Can Quietly Regress
Rephrasing a prompt to improve tone on 5 examples you tested can silently tank accuracy on 500 real-world cases you didn't. Without a baseline comparison, you'll never know until users complain.
Tested (5 examples)84% → 88% 🎉Real dist. (500 examples)84% → 54% 😱
03
New Model Versions Break Things You Never Tested
gpt-4o to gpt-4o-mini, Claude 3.5 to Claude 3.7 — each release brings subtle behavioral changes. JSON format drift, instruction-following changes, refusal behavior updates. Your eval suite catches this before prod does.
model: gpt-4o92%model: gpt-4o-mini (new)70% ⚠▶ Regression detected before deploy
04
Without a Baseline, You Can't Measure Progress
Teams that operate on vibes can't answer: “Did last sprint's changes help or hurt?” You need a frozen golden dataset and a baseline score to know if you're moving the needle — or just rearranging deck chairs.
Without baseline: ¯\_(ツ)_/¯With baseline: clear signal
Architecture

The 4 Layers of a Real Eval Harness

A production eval harness is not a single script. It's a stack — each layer serves a different purpose and runs at a different cadence. Build all four, and you have real confidence.

1
Foundation
Golden DatasetRequired
Representative inputs paired with expected outputs — the bedrock everything else runs against. Curate from real prod traffic, cover all capability categories, freeze it after initial creation. Add to it from production failures. Without this, no eval is meaningful.
2
Per-Capability Testing
Unit EvalsRun on PR
One test per atomic capability: classification accuracy, extraction F1, groundedness score, format compliance. Fast to run (<60s), easy to isolate failures. Block merges that drop any metric below threshold. These are your unit tests — except for prompts and model behavior.
3
Deploy Gate
Regression SuiteEvery Deploy
Runs the full golden dataset on every candidate deploy. Compares against the last known-good snapshot. If more than N% of previously-passing cases now fail, the deploy is blocked automatically. This is your airbag. It fires silently until you need it — then it saves you.
4
Live Signal
Shadow EvaluationA/B on Prod
Route a percentage of real traffic through the new prompt or model in shadow mode. Score both responses with LLM-as-judge without showing users. After collecting statistical significance, compare distributions. If the new version wins, ship. If it doesn't, rollback costs zero.

The key insight: each layer catches different failure modes. Unit evals catch "did I break this specific capability." Regression suites catch "did I break anything." Shadow evals catch "is this actually better in the real world." You need all three.

What to Measure

Metrics That Actually Matter

Different AI systems need different metrics. Here's the full breakdown — and a live eval report showing what good looks like.

📊 eval-report · run #1,847 · 2025-07-24 14:33 UTCCI GATE
RAG System
faithfulness0.91
answer_relevance0.88
context_recall0.74
hallucination_rate2.1%
Agent / Tool Use
task_completion87.3%
tool_accuracy93.1%
loop_detection100%
avg_steps_to_goal4.7
Classifier
precision0.94
recall0.81
f1_score0.87
false_negative_rate19%
run_id: r1847 · dataset: golden-v4 · 10,247 samples247/250 passed ✓  3 regressions ✗
📡
RAG Metrics

Faithfulness — is the answer grounded in the retrieved context, or did the model hallucinate?

Answer Relevance — does the response actually address the question?

Context Recall — did the retriever fetch the documents needed to answer? Low recall here means your retriever is the bottleneck.

🤖
Agent Metrics

Task Completion Rate — did the agent finish the goal without human intervention?

Tool Accuracy — did it call the right tool with correct parameters?

Loop Detection Rate — the 100% you always need. An agent stuck in a retry loop will burn your entire token budget.

🎯
Classifier Metrics

Precision — of what you flagged positive, how many actually were?

Recall — of all actual positives, how many did you catch?

Confusion Matrix — the full picture. Never optimize precision and recall as single numbers without looking at what the errors are.

✍️
Generation: Skip BLEU

BLEU and ROUGE measure n-gram overlap with a reference. They'll penalize a perfect paraphrase and reward a grammatically broken copy-paste.

Use LLM-as-judge instead — ask a stronger model to score coherence, helpfulness, and factuality. Correlates far better with human preference.

The Modern Approach

LLM-as-Judge

Use a stronger model to evaluate the outputs of a weaker one. It's scalable, aligns with human judgment, and can evaluate dimensions that heuristics never could — like tone, coherence, and factual precision.

Judge Prompt Pattern
🤖 Evaluated Model (e.g., GPT-4o-mini)
answers the user's question
Output: "The capital of France is Paris, which has been the capital since 987 AD and is home to approximately 2.1M people."
↓ + [original question] + [rubric]
⚖️ Judge Model (e.g., Claude Opus / GPT-4o)
evaluates against rubric
✓ Factual accuracy: PASS (5/5)
✓ Completeness: PASS (4/5)
✓ Hallucination: NONE detected
→ Overall: 9/10

Calibration tip: before trusting your judge, run 200 cases through it and compare against human labels. A well-calibrated judge should agree with humans at >85% rate. If it doesn't, adjust your rubric.

Prompting Patterns That Work
Use structured rubrics, not free-form
Give the judge explicit dimensions to score (accuracy, completeness, tone) with integer scales. Free-form “is this good?” produces high variance and poor calibration.
Force chain-of-thought before verdict
Ask the judge to reason step by step before giving a score. This dramatically reduces position bias and produces more consistent verdicts.
Known Pitfalls
⚠️
Verbosity biasLLM judges often score longer answers higher, independent of quality. Counteract with explicit rubric guidance: “length should not influence score.”
⚠️
Self-preferenceGPT-4 prefers GPT-4 outputs; Claude prefers Claude outputs. Use a different family model as judge, or use multiple judges and average.
⚠️
Position biasIn A/B comparisons, the judge often favors whichever response is listed first. Always run both orderings and average to eliminate this.
Close the Loop

The Eval Flywheel

The best eval harnesses get better over time — automatically. Every production failure feeds back into the golden dataset, making the next eval run sharper. This is the compounding advantage.

🚨PROD🗂GOLDENEVAL🚀SHIP
Prod Failure
Something breaks in the wild
Golden Dataset
Add the failure case
Eval Catches It
Next run detects regression
Ship Confident
Gate passes, deploy green
TheFlywheel

Compound advantage: a team that has been running this flywheel for 6 months has a golden dataset with hundreds of real-world failure cases — edge cases that no one thought to test upfront. Their eval suite is dramatically more powerful than a team starting fresh. Start the flywheel now.

The Takeaway

The teams shipping AI with confidence aren't guessing. They're measuring.

Stop relying on vibes. Build the golden dataset, wire up the eval layers, run the flywheel. That's the difference between shipping fearlessly and shipping anxiously.

Build Real Quality Gates for Your AI

Golden dataset → Unit evals → Regression suite → Shadow eval → Flywheel. That's the stack.