Open the model card for anything that shipped in the last month — Kimi K3, GLM-5.2, DeepSeek V4, Qwen 3.7 — and the coding section has quietly grown a new vocabulary. SWE-bench is still there, but it is no longer the headline. Above and around it sit names that did not exist eighteen months ago: DeepSWE, FrontierSWE, ProgramBench, SWE-Marathon, Terminal-Bench 2.1. A founder trying to pick a coding backend now faces six numbers where there used to be one, and no vendor explains why the six disagree.
They disagree on purpose. The useful way to read this stack is not to hunt for the biggest number — it is to notice that the benchmarks are arranged, whether or not anyone labels them that way, as a horizon ladder. Each one tests the agent at a different task length, and a model's scores don't rise or fall together. The gap between them is the signal.
Why SWE-bench stopped being the headline#
Two things happened at once. First, the frontier saturated it: once models cluster in the high 70s and 80s on SWE-bench Verified, the number stops separating them — everyone passes, so it decides nothing. Second, trust cracked. When OpenAI retracted roughly 30% of SWE-bench Pro as broken tasks, the community learned that a benchmark can be both widely cited and substantially wrong. SWE-bench survives as a floor — a model that can't clear it isn't a serious coding agent — but it moved off the marquee, and the new names moved on.
The ladder, one rung at a time#
Read the new benchmarks by their horizon, shortest to longest.
Terminal-Bench 2.1 is the near rung: a bounded task in a real shell — build this, debug that, make the tests pass — measured in minutes. It's honest about tool use and environment friction, but a high score here says nothing about stamina. It's the benchmark most like a good demo.
DeepSWE is the one worth understanding in detail, because it fixes the flaw everyone suspected in SWE-bench. It is 113 original tasks, written from scratch across 91 repositories and 5 languages, and its defining property is that it is contamination-free — none of the issues are lifted from public commits or pull requests, so a model earns nothing for having seen the patch during training. Its solutions require about 5.5x more code than a SWE-bench Pro task. Scores drop when you move a model from SWE-bench to DeepSWE, and that drop is not the benchmark being unfair. It is the memorization credit being removed.
FrontierSWE stretches the horizon again, from an hour toward multi-hour implementation projects — not "fix this issue" but "build this feature." SWE-Marathon is the far rung: it runs until the agent finishes or gives up, and it is explicitly a test of persistence and recovery — of what happens after the first three approaches fail. ProgramBench sits off to the side of the ladder, testing a different muscle entirely: infer and recreate a program's behavior with no source code in front of you. A repair score doesn't transfer to it.
Read the spread, not the peak#
Here is the number nobody quotes. On Moonshot's own reported card, Kimi K3 posts 88.3 on Terminal-Bench 2.1, 81.2 on FrontierSWE, 77.8 on ProgramBench, 67.5 on DeepSWE — and 42.0 on SWE-Marathon. That 46-point collapse from the near rung to the far rung looks like an embarrassment. It isn't. It is the most informative line on the page.
A model's best and worst coding benchmark, read together, draw a map of its failure horizon — the task length at which it stops being useful. The single tallest bar hides exactly the thing you need to know.
A model that scores in the 80s on bounded tasks and the low 40s on marathon tasks is telling you, precisely, that it is excellent for scoped fixes and unreliable for anything that has to run for an hour without a human. That is not a flaw in the reporting; it is the reporting working. The meta-skill of reading a coding-agent benchmark is learning to want the spread instead of the peak — the same move that reading an agent-memory benchmark requires, where the honest number is the one the vendor charts least.
What to do with six numbers#
Match the horizon to your workload and throw the rest away. If your agent does bounded, well-scoped fixes inside a known repo, weight Terminal-Bench and DeepSWE and stop there. If it carries multi-hour features, refactors, or anything that has to survive its own mistakes, weight FrontierSWE and — above all — SWE-Marathon, because that is where the gap between a launch demo and a Tuesday-afternoon production run shows up as a number. The benchmark closest to your real task is the only one whose score will transfer; the peak is marketing, and the saturated headline is theater.
The reason the model cards grew from one coding number to six is that "can it code?" was always the wrong question. The right one is "how long can it code before it falls off," and for the first time the cards answer it — if you read the collapse instead of the crest.
