The model judged it fine. That's not evidence.

You tweak a prompt, then ask a model which answers are better, the old batch or the new one. It says the new ones. You ship. Two days later the support queue fills with the exact failure you thought you fixed. The judge told you it was better and the judge was wrong, and you had no way to know because a preference from one model on a handful of cases was the whole basis for the change.

This is the most common way I see AI work go sideways: "the model judged it fine" gets treated as a result. It isn't one. A judge model is a model, with the same failure modes as the thing it's grading, plus a few of its own that have been measured. This module is about what does count as evidence, and it's cheaper to build than the debugging you'll do without it.

Guess how the judge is biased

Before the mechanics, three predictions. Each one is a real, measured bias in LLM-as-judge setups. Guess before you read the number.

Position, verbosity, self-preference. An unchecked judge launders these straight into your metrics, and the number you report looks rigorous while measuring the wrong thing. The same paper that named these biases also found something more hopeful: GPT-4's agreement with human labelers on MT-Bench was about 85%, which actually beat the human-human agreement of 81%. A judge can be genuinely useful. It just has to be checked first, the way you'd check any instrument before you trust its readings.

The golden set is the instrument

Before you can check a judge, or a prompt, or a model swap, you need something to check it against. That's a golden set: a small collection of real cases where you already know the right answer, assembled for your exact task.

I wrote a whole field note on one of mine in the newest model lost, where a golden set caught the newest, biggest vision model losing to an older, cheaper one on a real reading task. The shape is always the same. Twenty or thirty real cases, hand-labeled once, and "this feels better" becomes "this is six points better on the same cases, and here are the four that moved." The cases don't have to be many. They have to be real, and the labels have to be right, because a wrong label corrupts every score you read after it.

The order matters too. Read the actual outputs first, before you invent metrics. Pull thirty real responses, label each as pass or fail, and write down why each failure failed. Group the reasons. That grouping is your eval spec, because it's your product's actual failures rather than the ones a generic benchmark imagines. Metrics come after you know what you're counting.

Check the judge before it grades

Once you have labeled cases, an LLM judge stops being an act of faith. Run it against the same cases a human already labeled and measure how often it agrees. If it matches your human labels 90% of the time, it's an instrument you can point at the unlabeled pile. If it matches 60%, it's noise wearing a lab coat.

Two rules make judges more trustworthy. Prefer a binary pass or fail over a 1-to-5 score, because "why is this a 3 and not a 4" is a question neither the model nor you can answer consistently. And report the judge's agreement as two numbers, not one, which is the same distinction the next section is about.

Precision and recall, never bare accuracy

Say you build a detector that flags hallucinations in generated answers. You can tune it two ways. Tune it for recall and it flags almost everything, catching every real hallucination and a pile of fine answers with it. Tune it for precision and it only flags what it's sure about, missing the subtle ones but rarely crying wolf.

Which way you tune depends on the cost of each mistake. My citation verifier, verify-kit, is tuned hard for precision on the abstention side: when it says a claim is unsupported, that had better be true, because a false accusation erodes trust faster than a miss. Report both numbers and the tradeoff is legible. Report "accuracy" and you've hidden it.

Why ten examples tell you almost nothing

Here's the one that catches careful people. You run your new prompt on ten cases and it scores 7 out of 10. Feels like 70%.

It isn't 70%, it's "somewhere between roughly 40% and 90%, with 70% as the midpoint." That's the 95% confidence interval on 7 out of 10. The interval is that wide because ten coin-ish flips just don't pin down a rate. A system that's truly 70% and a system that's truly 90% will frequently produce the same score on ten examples, which means a ten-case eval often can't tell a real improvement from noise. If you ship on that, you're shipping on a feeling with a number stapled to it.

The fix isn't fancy. It's more cases. The narrower interval you want, the more labeled examples it costs, and knowing that ahead of time is the difference between "we detected a two-point gain" being real or imaginary. A good habit: never report a score without the count of cases behind it, so anyone reading it can feel how firm the ground is.

In production: the eval comes first

Across the tools I've built, the eval is load-bearing exactly because it exists before the change. On PermitCheck, the golden set of real decided permit applications scored every model swap and prompt tweak, and it's what surfaced that a batch of confident failures were actually a bug in my own rules. On verify-kit, the NLI and judge components that decide whether a paper supports a claim are themselves graded against a labeled set, because a grounding checker that isn't measured drifts toward agreeing with whatever it's handed. I wrote about paying two frontier models to break that checker in same words, opposite claim, and both breaks were only visible because the eval caught the checker saying yes when the honest answer was no. The NoeticMap pipeline that extracted structured claims from tens of thousands of papers runs the same way: a small graded sample gates every schema change. None of these would be trustworthy on vibes. They're trustworthy because a measurement was built before it was needed, and the whole cost was thirty labeled cases and the discipline to read them.