Most coding-agent benchmarks measure the model. SWE-bench and its cousins hand an agent a repo and a failing test and count fixes. Useful — but they tell you almost nothing about whether an agent can stand up your backend: write a migration that applies, ship an Edge Function that runs, or author an RLS policy that actually blocks the read it's supposed to block. On August 1, 2026, Supabase shipped a benchmark that measures exactly that, and open-sourced the whole thing.
What it is, in one screen#
supabase/evals (Apache-2.0, github.com/supabase/evals) runs AI coding agents against real Supabase tasks and scores what they do. The agents that ship as adapters at launch are Claude Code, Codex, and OpenCode. The tasks are the ones that actually hurt when they go wrong: build a schema, debug a failed Edge Function, fix a broken RLS policy.
The word doing the work is real. Every scenario boots a hosted-like Supabase stack and a local CLI project — both in containers, neither a mock — so the agent operates against a live backend. Scoring then mixes deterministic checks (did the migration apply? does the policy block the unauthorized query?) with an LLM-as-judge for the parts that resist a boolean. Each agent gets one retry. Results land as local files under results/ and feed the public leaderboard at supabase.com/evals.
The taxonomy is worth reading, because it's how you'll slice your own runs. Three axes:
- Products — database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api
- Topics — RLS, security, migrations, SQL, SDK, observability, self-hosting, tests, declarative-schema
- Stages — build, deploy, investigate, resolve
That last axis matters more than it looks. "Build" is the demo-friendly case — start from nothing, write the thing. Investigate and resolve are the on-call cases: something is already broken, and the agent has to find and fix it against a live stack. Those are the tasks that separate a slick autocomplete from something you'd let near production.
The finding: the model wasn't the variable#
Here's the part to cite. In the launch snapshot's Build stage, Opus 5 and Kimi K3 both scored 100% unaided. The rest of the field didn't get there by swapping to a bigger model — it got there with context files:
Sonnet 5 rose from 78% to 100%, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89% — once each was handed Supabase's own skills.
Read that twice. Giving the agent the vendor's context closed an 11-to-22-point gap. A cheaper model plus good skills matched an expensive model running bare. If you've ever watched an agent confidently write an RLS policy that's subtly wrong — or wondered why the skill you shipped never fired — this is the empirical version of that intuition: on a specific stack, the ceiling is set by what the agent knows about your stack, not by which frontier weights you rented.
That reframes a decision founders make constantly. The Cursor-vs-Claude-Code-vs-Copilot argument usually turns on the model behind the tool. Supabase's numbers suggest the higher-leverage question is: does this agent load my platform's context, and how good is that context? Spend on skills before you spend on tokens.
How to run it against your own agent#
It runs locally through pnpm. You compose experiment × eval pairs — an experiment is an agent-plus-config, an eval is a task — and each flag takes repeated or comma-separated values:
# clone + install
git clone https://github.com/supabase/evals && cd evals
pnpm install
# run two experiments against two specific tasks
pnpm eval -- \
--experiment claude-code-opus-5 \
--experiment claude-code-sonnet-5 \
--eval resolve-dataapi-001-empty-results \
--eval investigate-auth-001-deleted-user-access
# results are written under results/Because adapters are pluggable and the tasks are just data, the framework is only incidentally about Supabase's own leaderboard. The real use is regression-testing your agent setup: pin the tasks your product depends on (the migration pattern you use, the exact RLS shape your multi-tenant schema needs), add your agent as an experiment, and run it on every model bump. Supabase says it does exactly this — the same suite that powers the public page runs internally as a daily regression check. That's the pattern to copy: a benchmark you control is how you notice a model update quietly regressed your backend before your users do.
What to actually do this week#
- If you build on Supabase: clone the repo, run the
investigateandresolvetasks against the agent-plus-model you ship with, and look at where it fails without the context files. That delta is your skills backlog. - If you're choosing an agent: don't read the leaderboard as a ranking. Read it as proof that context closes most of the gap — then weight your choice toward the tool that best loads your platform's skills, not the one with the biggest model.
- If you build a dev tool: this is the template for a trustworthy benchmark — real environments, deterministic-plus-judge scoring, one retry, open tasks. "Runs against a live stack, not a mock" is now the bar. Benchmarks that run in a sandbox against fixtures will read as the weaker claim.
The launch scores will move — AI tooling always does, so treat the numbers here as a dated snapshot and check supabase.com/evals for live results. The structural lesson is the durable part: when the frontier models cluster at the top, the thing that still varies is what you tell them about your stack.
