You changed the system prompt. The demo looks better. You ship it — and three days later a customer forwards a screenshot of your agent confidently doing the wrong thing on an input you never tried. You have no way to know whether last week's "improvement" caused it, because "better" was a feeling, not a measurement.
That gap is what Braintrust is built to close. Its one idea, borrowed straight from software engineering: an eval is a test. You write it once, against real examples, with a scorer that returns a number — and then every prompt tweak, model swap, or agent change runs against it before it reaches a user. Instead of shipping on vibes, you ship on a diff in a score.
What it is: an eval-first LLM observability platform — founded by Ankur Goyal — that unifies three things founders usually bolt together from spare parts: evaluations (datasets + scorers), a playground for iterating on prompts and models side by side, and production tracing so the same scoring you run in CI also watches live traffic. It runs on Brainstore, a purpose-built store the company says queries AI traces far faster than a general-purpose database.
What it does#
- Evals as datasets + scorers. You collect real inputs (and expected behaviors) into a dataset, attach scorers — deterministic code checks, or LLM-as-judge grading for open-ended output — and run. The result is a score you can compare across versions. This is the piece that turns "I think it got better" into "regression on 4% of the eval set."
- A playground for the inner loop. Try a new prompt or a cheaper model against the same dataset and see the score and the diffs immediately, before you touch code. It pairs naturally with shadow-testing a cheaper model before you switch — the eval set is the referee.
- Loop, the built-in agent. Loop can generate test cases from your data, run evals, and iterate on prompts on its own — automating the tedious half of the trace → evaluate → improve cycle. There's also a Braintrust MCP server, so a coding agent like Claude Code can pull your evals and traces directly into its context.
- CI gating. The same evals run in your pipeline, so a pull request that drops the score can fail the build — the AI-native version of "tests must pass to merge."
Who it's for#
Founders and teams who've shipped an AI feature and hit the wall where "eyeball the output" stops scaling — usually the moment you have more than a handful of prompts, or a second person changing them. It earns its place fastest if you're about to swap models to cut cost, you keep re-introducing the same failure, or you need to show a customer (or yourself) that quality is going up and not sideways. Braintrust names Notion, Replit, Cloudflare, Ramp, and Dropbox among its users, and raised an $80M Series B led by ICONIQ at an $800M valuation in February 2026 — this is a category bet, not a side project.
How to start#
Sign up for the free Starter tier (no credit card), grab your API key, and wire the SDK into your app so calls start tracing. Then do the one thing that makes Braintrust click: create a small dataset from a dozen real inputs, attach one scorer, and run an eval. Seeing a single number move when you change a prompt is the whole pitch, and you can get there in an afternoon. From there, add the scorer to CI so a bad change can't merge.
What it costs (July 2026)#
- Starter — free. 1 GB of processed data, 10,000 scores, 14-day retention per month, no credit card. A genuine workbench for building your first eval suite.
- Pro — $249/mo, flat. 5 GB of data, 50,000 scores, 30-day retention, and — notably — unlimited users at every tier, so there's no per-seat tax as your team grows.
- Enterprise — custom. This is where self-hosting, SAML SSO, custom RBAC, HIPAA BAA, S3 export, and an uptime SLA live.
- Overage: $3 per additional GB and $1.50 per 1,000 scores.
The catch#
Three honest ones. First, Braintrust is closed-source — unlike Langfuse (MIT) or Arize Phoenix (OpenTelemetry-native), you can't self-host it without an Enterprise contract, so you're betting your eval history on a vendor. Second, billing is usage-based on data and scores, and a chatty production app that traces everything can climb the tiers faster than a flat-fee tool like Langfuse's $29 Core — model your volume before you rely on it. Third, the jump from free to $249 is steep for a solo builder; the Starter tier is a real place to build and validate an eval suite, but it isn't where a growing product lives.
If those are dealbreakers, the open-source stack — Langfuse or Phoenix, compared against managed options in our observability shootout — is the alternative. If they're not, Braintrust's bet is that the fastest way to ship AI you trust is to make evals a first-class object with the same weight your tests already carry — and for a team past the eyeball stage, that discipline is worth paying for.
