On July 8, OpenAI published an audit of SWE-Bench Pro — one of the most-cited agentic coding benchmarks, built by Scale AI — and did the rare thing: it retracted its own recommendation to use it. The headline number is that roughly 30% of the benchmark's 731 public tasks are broken. Not "hard." Broken — in the specific sense that a correct solution can be marked wrong.
If you route your coding agent by looking at where models land on a leaderboard, this is the memo you need. The short version: the two-point gaps that model launch posts wave around are smaller than the benchmark's own error.
What "broken" means here#
OpenAI ran two independent passes. An automated datapoint-analysis pipeline — which reads each model's attempt, the task metadata, and the failure trace to spot likely grading flaws — flagged 200 tasks (27.4%). A parallel human review by experienced software engineers flagged 249 (34.1%). Two methods, one conclusion: about a third of the set doesn't measure what it claims to.
The defects cluster into two shapes, and both are worth understanding because they'll show up in any benchmark you trust.
The largest category is overly strict tests. The hidden test suite asserts a specific implementation — a particular function signature, an exact output format, an internal data structure — that the task prompt never asked for. A model writes code that is functionally correct, passes any reasonable human review, and solves the stated problem. The test fails it anyway, because it didn't match the shape the benchmark author happened to write. The model gets a zero for being right in a different way.
The second is underspecified prompts: the task description omits a requirement, but the hidden tests enforce it. The model can't pass because it was never told the rule it's being graded against. This is the inverse failure — the benchmark knows something it didn't say.
A miscalibrated benchmark doesn't make a bad model look good. It makes a good model look unpredictable — and it drowns small differences between models in noise.
Why this changes how you route, not whether you route#
The wrong lesson is "benchmarks are theater, ignore them." They aren't, and you can't. The right lesson is about resolution.
Think of SWE-Bench Pro as a ruler with a third of its tick marks in the wrong place. It's still fine for telling a 30cm object from a 3cm one — a genuinely weak model from a genuinely strong one. It is useless for telling 30.0cm from 30.2cm. And 30.0-versus-30.2 is exactly the comparison the current frontier forces on you: the top coding models now sit within a few points of each other on most suites. When the ruler's error is larger than the gap you're trying to read, the gap tells you nothing.
So the founder move is a two-tier eval:
- Use public benchmarks as a first-round filter. They reliably eliminate models that are clearly not ready. If something is ten points off the pack, believe it.
- Decide between the finalists on your own tasks. Pull 20–50 real problems from closed pull requests in your own repo — bugs you actually fixed, features you actually shipped. Grade each model's diff by one question: would you merge it? That's a benchmark that grades on your bar, your stack, and your review standard, not Scale's. It takes an afternoon to build and it's the only score that predicts your experience.
The pattern under the retraction#
This is not the first time OpenAI has stepped back from a coding benchmark it once leaned on — it did the same with SWE-Bench Verified. (If you want the lineage of these suites, we walked through SWE-Bench Pro versus SWE-Bench Verified when Pro launched.) Read the two retractions together and the trend is clear: the field's headline coding benchmarks are decaying faster than credible replacements arrive. Models are now good enough that the benchmarks' own defects — the strict tests, the missing requirements, the tasks that leaked into training data — are a larger source of score variation than the models are.
A lab retracting a widely-cited, vendor-neutral benchmark is a much stronger signal than any lab boasting about its own number. Take the signal. The leaderboard is a filter now, not a verdict. The verdict is the eval you build from your own merged code — and the afternoon it costs is cheaper than a quarter spent on the wrong model because two points of noise pointed you there.
