Two numbers get quoted when someone says a model is "good at coding." One is its SWE-bench Verified score. The other is its Aider Polyglot score. They are treated as interchangeable evidence for the same claim, and they are not. They measure different skills, reward different models, and answer different questions. Quoting one when you needed the other is how teams end up with a coding agent that aces the demo and stalls on the job.
Here is the distinction, stated plainly, because almost no leaderboard states it.
SWE-bench Verified: can it work in my repo?#
SWE-bench Verified is 500 tasks, each a real GitHub issue paired with the pull request that fixed it, drawn from a handful of large, popular Python projects. Human annotators combed the original SWE-bench test set and kept only the problems that were unambiguously specified and genuinely solvable, so a failure means the model failed — not that the task was broken.
A passing run demands two things in sequence: the model has to find the bug inside a codebase far larger than its context window, then produce a patch that makes a set of hidden tests pass. That first step — localization at repo scale — is most of the difficulty, and it is the skill that matters when your agent lives inside an existing 400,000-line service. (It is also why the harder SWE-bench Pro cut exists: as models saturate Verified, the pressure moves to bigger, messier repos.) The number is honest about one thing and silent about another: it tells you the model can navigate and patch Python, and it tells you nothing about any other language, because there is no other language in the set.
Aider Polyglot: can it write correct code and apply the edit?#
Aider Polyglot is built on a different foundation. It takes 225 of Exercism's hardest exercises and spreads them across six languages — C++, Go, Java, JavaScript, Python, and Rust. The problems are self-contained; there is no repo to navigate. What replaces navigation is three things stacked on top of "solve it," and those three are the whole point.
First, the model must return its solution in a required edit format — a unified diff, a whole-file rewrite, or an "architect" plan-then-edit pass. A model that reasons its way to the right code but emits a malformed diff fails the task. That sounds pedantic until you remember it is precisely how real coding agents die: the model knew the fix and produced an edit the tooling couldn't apply.
Second, the model gets a second attempt. When the first solution fails, Polyglot hands back the failing unit-test output and lets the model try again; the headline score is the pass rate after that second try.
Aider Polyglot is not a code-generation benchmark wearing a leaderboard. It is a self-correction benchmark — it grades the debugging loop, and folds edit-application into the grade.
Third, the problem set is self-calibrating. When the original Python-only aider benchmark saturated — the top models simply solved all of it — the maintainers rebuilt it by keeping only the problems that three or fewer models could solve. That filter is a small, clever idea with a large consequence: the benchmark deliberately discards what everyone already gets right, so it stays hard as the frontier moves. It is a moving target by construction.
Why the gap between the two scores is the useful part#
Because the two benchmarks reward different skills, a model's rank can move between them — and that movement is diagnostic, not noise.
The single-language design of SWE-bench is the clearest example. Every task is Python, so a model's Rust or C++ ability is structurally invisible on it. A model can post a strong SWE-bench Verified number and still be the weakest thing you could put on a Go codebase, and you would never see it coming. Polyglot's six-language spread is the only place that variance shows up. Ask any vendor for the per-language Polyglot breakdown and watch how often the aggregate was carrying a soft language on its back.
The self-correction axis cuts the other way. A model that is brilliant one-shot but poor at reading a stack trace will look better on a single-patch benchmark than it deserves to, and worse on Polyglot's second-attempt scoring — which is the honest one, because your agent will always be on its second, third, and fourth attempt in production. The delta between first-attempt and post-feedback pass rates is the closest thing we have to a "debugging" number, and reporting only the final figure erases it.
What to actually do with this#
Stop asking "which model is best at coding." Ask which job you are hiring for.
- Agent that works inside a large existing repository? Weight SWE-bench Verified. You are buying localization and patching at scale, and that is what it measures. Discount Polyglot's algorithmic difficulty; your agent rarely writes a novel sorting routine.
- Agent that writes multi-language greenfield code and applies its own edits? Weight Aider Polyglot — and demand the per-language table and the first-attempt score. You are buying correctness across languages plus a clean edit-and-recover loop, and the topline number hides both.
And when neither fits — because your agent's job is running commands, not editing files — reach for a third axis entirely, like Terminal-Bench, which grades end-to-end shell work. Neither Polyglot nor SWE-bench is wrong. Each is honest about the skill it grades and silent about the one it doesn't. The mistake is reading a single number as a verdict on "coding," when the two most-quoted numbers in the field were never measuring the same thing.
