A cheap model "beats the flagship on nine agent benchmarks" roughly once a week now. Most of those claims are true in the narrow sense that the vendor's harness produced those numbers, and useless in the sense that matters — whether the model will beat your incumbent on your workload. Here's the five-question checklist to run on any agent-benchmark table before it changes your architecture, worked live on the freshest example: DeepSeek's V4-Flash-0731 release on July 31.

The short version: A vendor benchmark is a claim about the vendor's test harness as much as their model. Run five questions — who ran it, is the harness public, what baseline did they pick, is the metric the one your budget cares about, and has anyone independent reproduced it. If the answers are shaky, the number is a hypothesis to test, not a result to buy.

1. Who ran it?#

The lab that trained the model usually also built and ran the eval. That's not fraud; it's the default, and it's why a self-reported score starts at "unrefereed," not "wrong." DeepSeek's V4-Flash table is vendor-run. So is almost every launch-day table. The question isn't whether the vendor ran it — they did — it's whether you can check their run. Which leads straight to:

2. Is the harness public?#

This is the one that separates a real claim from a press release. DeepSeek's headline is DeepSWE jumping 7.3 → 54.4 — a ~645% move. It's also run on DeepSeek's own harness, which the changelog says will be released "soon." Until it ships, nobody outside the lab can tell whether the model got dramatically better or the test changed — a friendlier scaffold, a longer step budget, a re-scored rubric. A huge jump plus a closed harness is the single loudest red flag on any table. Not disqualifying. Just not bankable.

3. What baseline did they pick?#

Beating your own weakest prior checkpoint is easy to engineer. DeepSeek compares Flash-0731 against V4-Pro-Preview — its own earlier model — and wins on all nine. Interesting, but a preview is a soft target. The comparison that would move you is Flash-0731 against a current frontier model you don't control, on a public harness. DeepSeek gestures at this once — Agents' Last Exam 25.2 vs Opus-4.8's 25.7 — and notably that's the row where the margin nearly vanishes. When the only benchmarks a model wins are the ones baselined against the vendor's own back catalog, you're reading a story about their release cadence, not their capability.

4. Is the metric the one your budget cares about?#

Headline agent scores are pass-rates on someone else's suite. Your P&L runs on cost-per-completed-task on your workload. A model that scores 10% lower at a third of the price can still be the correct switch — the exact lens we used comparing Qwen3-7-Flash and Gemini 3.6 Flash. Notice which number on the DeepSeek table you can actually verify: the $0.14 / 1M input price. That's the real news most weeks — the price cuts are verifiable; the agent scores are not. Trust the price, test the benchmark.

5. Has anyone independent reproduced it?#

The last gate is time. When DeepSeek's harness ships and third-party runs land — the kind of cross-model work sites like Artificial Analysis do — the 54.4 either holds or it doesn't. Standardize after that, not before. Independent reproduction is the difference between a leaderboard result and a launch tweet, and it's the same discipline we applied to agent-memory benchmarks and to why one tokens-per-second number is lying to you.

The move#

Don't argue with the table. Reproduce it small:

  1. Pull two or three tasks from your own product — your repo, your tool chain, your prompts. Public suites rarely predict your failure modes.
  2. Run the candidate and your incumbent on the same tasks. Score cost-per-completed-task.
  3. Decide on that, and re-check when the vendor's harness and independent runs land.

The cheap-model floor is going to keep dropping and the launch tables are going to keep claiming frontier-level agent scores. That's fine — cheaper capable models are the best thing to happen to a solo founder's budget in two years. Just keep your own eval harness warm. It's the only benchmark that gets a vote on your production traffic.