The line in almost every eval writeup reads the same way: our LLM judge agrees with human labelers 80% of the time. It sounds like a measurement. It's actually two different questions wearing one number, and the number can't answer either of them cleanly.
The two questions are reliability — does the judge give the same verdict when you run it again — and validity — is that verdict correct. Psychometrics has kept these apart for a century because they fail independently: a bathroom scale that always reads five pounds light is perfectly reliable and completely invalid. In 2026 we handed grading to a stochastic model and quietly assumed the two axes had merged. They haven't.
The coin flip you never saw because you ran it once#
Start with reliability, because it's the one people assume they already have. A June 2026 study bluntly titled The Coin Flip Judge? took 29 tasks across ten categories, ran two OpenAI judges over each item 50 times pairwise and 50 times pointwise, and simply looked at how often the verdict held.
It didn't. Pairwise preferences flipped 13.6% of the time on average. On 28% of questions the flip rate cleared 20%; one question flipped 56% of the time — a literal coin. This is not two judges disagreeing. It's one judge disagreeing with itself on identical input.
The verdict flip rate is a number you can only see by rerunning. Run the eval once and you didn't measure the judge — you drew one sample from a distribution you never looked at.
The sharpest finding is what sat underneath the flips. The judges' own pointwise scores for the two answers differed by just 0.19 to 0.36 points on a 10-point scale, and the gaps weren't statistically significant. So the model's scalar judgment said these answers are indistinguishable — and the pairwise prompt then forced it to crown a winner anyway. The decisiveness in "A beats B" is manufactured by the question format, not earned by the evidence. Deterministic decoding narrows the gap; it does not close it. A corroborating paper, Rating Roulette, found the same self-inconsistency across frameworks.
The other failure: a judge that's steady and wrong#
Now flip to validity, and you find the opposite pathology hiding in plain sight. The largest audit to date, Reliability without Validity, put 21 judges from nine providers through MT-Bench, JudgeBench, and RewardBench — 118 runs, roughly 541,000 individual judgments. Two of the judges, both in production somewhere, scored test–retest reliability above 0.95 while carrying position bias above 0.10.
Read that pairing slowly. Above 0.95 reliability means the judge is nearly deterministic — rerun it and you get the same answer. Above 0.10 position bias means that answer partly reflects which option was listed first, not which was better — one of the systematic biases (position, verbosity, self-preference) that shift verdicts in a fixed direction rather than at random. The judge is repeatable and biased at the same time. Its stability is exactly what makes the bias dangerous: a needle that never moves reads like a trustworthy instrument, even when it's pointing at the wrong number.
The headline stat is inflated before you even start#
Even the "80% agreement" itself is a soft number. Exact-match agreement doesn't correct for chance — two coins agree half the time for free. When the same audit computed chance-corrected agreement (Cohen's κ), raw exact-match overstated it by 33 to 41 percentage points on MT-Bench. Your proud "agrees 80% of the time" can be a κ around 0.45, which is "moderate" and a long way from "ground truth." Rankings built on the inflated metric aren't stable either: judge order shifted by up to 14 positions depending on which benchmark you scored on — the same way a confidence interval quietly swallows a model leaderboard once the gaps shrink below the noise.
Why "pin the judge" is comfort, not a cure#
The standard hardening advice — pin the judge model, set temperature to 0 — is real and worth doing, but notice which axis it touches. Pinning and temperature 0 buy you reliability: less drift, less run-to-run variance. That was rarely the axis quietly poisoning your numbers. Neither move does anything for validity — a pinned, deterministic judge with self-preference bias is just a reproducibly self-preferring judge. Worse, the low-variance dashboard it produces looks more credible, so a validity problem gets easier to miss precisely as you "fix" reliability.
What to actually report#
The discipline here is boring and cheap:
- Rerun and publish the flip rate. N reruns of the same eval, report how often verdicts change. If it's north of 10%, your single-run scores are noise dressed as signal.
- Report κ, not raw agreement. Chance-corrected or it doesn't count. A number that can't beat two coins agreeing isn't evidence.
- Randomize answer order and measure the position-flip rate. That's your cheapest validity probe; a judge that changes its mind when you swap A and B is telling you what it's really scoring.
- Distrust pairwise winners without a significant pointwise gap. If the judge's own scores can't separate the answers, the "winner" is the format talking.
- Treat safety-eval judges as hypotheses. On a 6,442-sample human-verified robustness benchmark, state-of-the-art judges came in no better than a coin flip once red-teaming shifted the distribution. There, the judge isn't a ruler — it's a rumor.
None of this makes LLM judges useless. It makes them instruments, and instruments come with error bars. The failure mode isn't using a model to grade a model. It's reporting a single draw from a distribution you never characterized, calling it 80%, and shipping the decision on top of it.
