The one-screen answer: To pick which model runs your coding agent, don't read the leaderboard — build a 20–50 task eval from your own closed issues and run each candidate through your own agent scaffold. For every task: checkout the pre-fix commit, hand the model the issue, apply its patch, run your repo's tests, and log pass/fail + cost + wall-time. Then rank by cost-per-solved-task and reliability in the loop, not raw resolve rate. It's one afternoon, and it measures the one thing SWE-bench and Terminal-Bench structurally can't: how a model behaves on your code, in your harness, at your cost ceiling.

This is the concrete follow-up to how to read a coding-agent benchmark. That piece's whole argument was that a public score is a model + scaffold + task-set bundle, and you only control one of the three. The fix is to control all three yourself. Here's the code.

Step 1: Harvest 20–50 real tasks from your repo#

A task is a closed issue or PR that shipped with a test that fails before the fix and passes after. The test is your oracle — no test, no task. Harvest from merged PRs that touched your test directory:

# Pull merged PRs that changed tests, newest first. Needs `gh` + `jq`.
gh pr list --state merged --limit 200 \
  --json number,title,mergeCommit,files,closingIssuesReferences \
  --jq '[.[] | select(any(.files[].path; test("tests?/|_test\\.|\\.test\\."))) ]' \
  > candidate_prs.json

# Eyeball it, then keep 20–50 where the issue body is self-contained.
jq 'length' candidate_prs.json

For each PR you keep, you need three things: the parent commit (the repo state before the fix), the issue text (what the model gets to see), and the test command that goes red→green. Build a small tasks.jsonl:

# harvest.py — turn kept PRs into eval tasks
import json, subprocess

def parent_sha(merge_sha: str) -> str:
    # first parent = mainline commit the fix landed on top of
    out = subprocess.run(["git", "rev-parse", f"{merge_sha}^1"],
                         capture_output=True, text=True, check=True)
    return out.stdout.strip()

tasks = []
for pr in json.load(open("candidate_prs.json")):
    issue = (pr["closingIssuesReferences"] or [{}])[0]
    tasks.append({
        "id": f"pr-{pr['number']}",
        "base_commit": parent_sha(pr["mergeCommit"]["oid"]),
        "problem": issue.get("body") or pr["title"],   # what the model sees
        "test_cmd": "pytest -q tests/",                 # your real suite (or a targeted subset)
        "gold_files": [f["path"] for f in pr["files"]], # for failure analysis only
    })

with open("tasks.jsonl", "w") as f:
    for t in tasks:
        f.write(json.dumps(t) + "\n")
print(f"wrote {len(tasks)} tasks")

Two rules that save you pain: never expose gold_files to the model (that's answer-leakage), and exclude tasks that need external infra your harness can't stand up — pure dependency bumps, secret rotations, anything requiring a live third-party service. This is a specialized case of building an eval dataset; the general principles are in how to build an LLM eval dataset.

Keep these tasks private. They're your repo's real issues. Don't push tasks.jsonl to a public bucket — you don't want your eval leaking into a future model's training set, which is the same contamination problem that erodes public benchmarks.

Step 2: Build the tiny harness#

The harness does the same four moves for every (task, model) pair. The key is that run_agent() calls your actual agent scaffold — the same prompt, tools, and turn limit you run in production — not a clean-room reimplementation. If your real loop is different from the eval loop, you're back to measuring someone else's harness.

# run_eval.py — one candidate model over all tasks
import json, subprocess, time, tempfile, shutil, os
from your_agent import run_agent   # <-- YOUR real scaffold: (repo_dir, problem, model) -> {diff, cost_usd, tokens}

REPO = "/path/to/your/repo"

def checkout(dst: str, sha: str):
    shutil.copytree(REPO, dst, dirs_exist_ok=True)
    subprocess.run(["git", "-C", dst, "reset", "--hard", sha], check=True,
                   capture_output=True)
    subprocess.run(["git", "-C", dst, "clean", "-fdx"], check=True,
                   capture_output=True)

def tests_pass(dst: str, cmd: str, timeout=900) -> bool:
    try:
        r = subprocess.run(cmd, cwd=dst, shell=True, timeout=timeout,
                           capture_output=True)
        return r.returncode == 0
    except subprocess.TimeoutExpired:
        return False   # a wedged/hung run is a FAIL, not a skip

def run_task(task: dict, model: str) -> dict:
    dst = tempfile.mkdtemp(prefix="eval-")
    rec = {"id": task["id"], "model": model, "solved": False,
           "cost_usd": 0.0, "wall_s": 0.0, "failure": None}
    try:
        checkout(dst, task["base_commit"])
        t0 = time.time()
        try:
            out = run_agent(dst, task["problem"], model)   # your loop runs here
        except Exception as e:
            rec["failure"] = f"agent_error:{type(e).__name__}"
            return rec
        rec["wall_s"] = round(time.time() - t0, 1)
        rec["cost_usd"] = out.get("cost_usd", 0.0)

        patch = out.get("diff", "")
        if not patch.strip():
            rec["failure"] = "empty_patch"; return rec
        ap = subprocess.run(["git", "-C", dst, "apply", "--3way"],
                            input=patch, text=True, capture_output=True)
        if ap.returncode != 0:
            rec["failure"] = "patch_did_not_apply"; return rec

        rec["solved"] = tests_pass(dst, task["test_cmd"])
        if not rec["solved"]:
            rec["failure"] = "tests_red"
        return rec
    finally:
        shutil.rmtree(dst, ignore_errors=True)

if __name__ == "__main__":
    import sys
    model = sys.argv[1]                       # e.g. "claude-opus-5"
    trials = int(os.environ.get("TRIALS", 3)) # runs per task -> variance
    tasks = [json.loads(l) for l in open("tasks.jsonl")]
    with open(f"results-{model}.jsonl", "w") as f:
        for task in tasks:
            for _ in range(trials):
                f.write(json.dumps(run_task(task, model)) + "\n")

A few decisions in there matter. A timeout is a fail, not a skip — a model that hangs your loop is unreliable even when it's otherwise smart. Run each task 2–3 trials (TRIALS=3), because agent runs are stochastic and a single pass hides the variance that will actually bite you. And git apply --3way tolerates minor drift instead of failing on a one-line context mismatch. If you'd rather not hand-roll this, Inspect AI (dataset → Task → Solver → Scorer, with sandboxed execution) and the local-first Promptfoo both give you the scaffolding; the logic above is what they're doing under the hood.

Step 3: Run each candidate over the same tasks#

Same tasks, same harness, one model swapped at a time:

for m in claude-opus-5 gpt-5.6-sol kimi-k3; do
  TRIALS=3 python run_eval.py "$m"
done

Point each name at the right endpoint in your run_agent — Anthropic for Claude Opus 5, OpenAI for GPT-5.6 Sol, and Moonshot or a third-party host like OpenRouter for Kimi K3. Keep temperature and turn-limit identical across all three; you're comparing models, not prompts.

Step 4: Read it as cost-per-solved-task, not resolve rate#

Now roll up the results. The headline number is not resolve rate:

# score.py
import json, glob, statistics as st
for path in glob.glob("results-*.jsonl"):
    rows = [json.loads(l) for l in open(path)]
    solved = sum(r["solved"] for r in rows)
    spend  = sum(r["cost_usd"] for r in rows)          # ALL attempts, solved or not
    waits  = sorted(r["wall_s"] for r in rows)
    p95    = waits[int(0.95 * len(waits)) - 1]
    fails  = {}
    for r in rows:
        if not r["solved"]:
            fails[r["failure"]] = fails.get(r["failure"], 0) + 1
    print(f"{path}: resolve={solved/len(rows):.0%}  "
          f"$/solve={spend/max(solved,1):.3f}  "
          f"p95={p95:.0f}s  fails={fails}")

Read the output in this order:

  1. Reliability gate first. Throw out any model whose p95 latency or wedge rate (agent_error + timeouts) breaks your loop. A model that hangs on 1 in 10 unattended runs is disqualified regardless of how smart it is — reliability compounds hardest when a human isn't watching.
  2. Cost-per-solved-task second. $/solve counts every attempt, including the failures you paid for. A model at 78% and $0.11/solve beats one at 82% and $0.60/solve for anything running at volume — the unit economics are in what an AI agent costs per task.
  3. Resolve rate last, as a tiebreaker. Across 30 tasks a 78%-vs-80% gap is noise; treat it as a tie and let the first two axes decide.
  4. Read the failure modes. fails={'patch_did_not_apply': 6} means a formatting problem in your scaffold, not a dumb model. tests_red on the same 3 tasks across every model means those tasks are underspecified — fix the eval, not the model.

Once this passes, wire the winning model's task set into CI so a model or prompt change can't silently regress you — see how to add LLM evals to CI/CD and, because agent evals are stochastic, how to run agent evals in CI without a flaky gate.

Public benchmarks are a filter for who gets into this eval. They were never the decision. An afternoon of your own repo's issues tells you what a year of leaderboard-watching can't: which model actually ships your work, reliably, at a price you can pay.