If you read one line: A coding-benchmark score is three things bundled — the task set, the agent scaffold around the model, and the pass criterion — and changing any one of them moves the number more than the gap between two frontier models does. Read those three before you read the rank.

A new model claims #1 on a coding leaderboard almost every week. In July alone, Kimi K3 landed with a reported 88.3% on Terminal-Bench and a #1 slot on the Frontend Code Arena (~1,679 Elo), while Claude Opus 5 was cited at ~96–97% on SWE-bench Verified (Moonshot, Anthropic). All of those numbers can be true and none of them, on its own, tells you which model should run your agent. Here's how to read them so the leaderboard informs your choice instead of making it for you.

Rule 1: "SWE-bench" is not one number#

The most common founder mistake is comparing two SWE-bench scores that aren't the same test.

A 94% on Verified and a 55% on Pro describe the same capability at very different difficulties. If a launch post writes "SWE-bench" with no suffix, the number is unspecified — treat it that way. We walk the two apart in detail in SWE-bench Pro vs SWE-bench Verified.

Rule 2: a coding score is a model plus a scaffold#

This is the one that actually explains why leaderboards don't survive contact with your repo.

The number a vendor reports is not the model in isolation. It's the model wrapped in a scaffold — the agent harness that hands it the repository, decides how many turns it gets, lets it run tests and retry, and feeds errors back. The same weights can swing well over ten points on SWE-bench depending on that harness. Vendors, reasonably, report the score from their best-tuned scaffold. You will run a different one.

What it means: when you read "Model X scores 71% on SWE-bench," mentally append "…in the vendor's harness." Your number depends on your agent loop — which is why Terminal-Bench and SWE-bench measure different slices of the same skill, and why a fixed-harness benchmark like Aider Polyglot isolates the model better than an open-scaffold one.

Rule 3: match the benchmark to the job#

Different coding benchmarks measure different jobs, and a #1 on one says nothing about another:

Kimi K3 being #1 on the Frontend Arena and strong on Terminal-Bench are two separate facts about two separate jobs. Decide which job you're hiring for first.

Rule 4: at saturation, the points are noise#

When the top of a leaderboard clusters at 95%+, the remaining gap is dominated by test contamination, harness luck, and label noise — not capability. The difference between a reported 96% and 97% on SWE-bench Verified is inside the error bar. Treating it as a ranking is reading a benchmark the way it's built to be misread.

Read a saturated benchmark as a tier, not a rank: is this model in the frontier tier for my kind of work? If yes, the leaderboard has done its job — now let cost, latency, data control, and your own eval break the tie. That's the same logic behind why one tokens-per-second number lies to you and why the leaderboard has quietly become theater at the top.

What to actually do#

  1. Identify the job. Agentic multi-step work, single-issue fixes, or UI generation each have a different right benchmark.
  2. Read the scaffold and the suffix. Which SWE-bench? Which harness? Self-reported or independent?
  3. Treat single-source numbers as "as reported." One outlet, one run, no replication → provisional.
  4. Run a private eval. Pull 20–50 real issues from your own repository, wrap them in your agent scaffold, and run every finalist through. It takes an afternoon and it measures the only thing that matters: how the model behaves on your code, at your cost ceiling.

Public benchmarks are a filter for deciding who gets into that eval. They were never meant to be the decision — and once you read them as task-plus-scaffold-plus-criterion, the weekly "#1" headlines stop being confusing and start being useful.