The short version: VitaBench — an agent benchmark from Meituan's LongCat team, accepted at ICLR 2026 — drops an LLM agent into three messy real-world domains (food delivery, in-store ordering, online travel), hands it 66 tools, and gives it a user who is vague and keeps changing their mind. The best models on the market finish 32.5% of the cross-domain tasks. If your instinct is "so agents don't work," you're reading the number wrong. The 32.5% is the honest number, and the way it's low tells you exactly where to be careful.

What VitaBench actually tests#

Most benchmarks that produce comfortable 70%-plus scores test one clean task in one environment. VitaBench does the opposite on purpose. Its 100 cross-scenario tasks (the headline evaluation, backed by 300 single-scenario ones) each require the agent to:

That's not a coding task with a hidden unit test. It's the actual shape of the "just handle it for me" product every founder wants to build.

Why the score is low — and why that's the feature#

Two design choices drive the number down, and both make it more trustworthy.

The scoring is all-or-nothing. Each task has a hand-written rubric, and the agent succeeds only if it satisfies every item, across four runs at temperature 0.0. There's no partial credit. Order the right dish from the wrong restaurant and you don't get 80% — you get zero. That's brutal, but it mirrors reality: a booking agent that gets four of five details right still booked the wrong flight.

The tasks are hard where agents are weakest. The moment a job crosses domains and the user gets vague, models start guessing instead of asking, lose the thread of what the user reversed two turns ago, or call a plausible-but-wrong tool. VitaBench is engineered to catch exactly those failures.

A high score on a narrow benchmark tells you a model can do one thing. A low score on a realistic one tells you where it breaks. The second is the one you can act on.

The one number to take to your roadmap#

Forget the leaderboard ranking. The most useful figure in the whole paper is the collapse from under 62% on single-scenario tasks to 32.5% on cross-scenario ones.

That drop is your risk surface, drawn to scale. Agents hold together when the task lives in one domain with clean inputs — that's the ~62% world. They fall apart when the task spans systems and the user is ambiguous — the ~32% world. If your product deck quietly assumes an agent will happily orchestrate across three services in a single vague conversation, VitaBench is the free, independent warning that you are aiming at the hardest 32%, not the friendlier 62%.

This is the same reading discipline we've argued for elsewhere: a benchmark number is only meaningful once you know its scoring rule, which is the entire point of how to read a coding-agent benchmark and how to read an agent-memory benchmark. VitaBench sits in the tau-bench lineage of real-world interactive evals — same family, harder tasks.

What to actually do with it#

  1. Scope the first version to one domain. Ship into the ~62% region, not the ~32% one. A single-service agent that works beats a cross-service agent that fails a third of the time.
  2. Instrument ambiguity. Log every turn where the user's intent was unclear and whether your agent asked or guessed. VitaBench's whole thesis is that guessing is where tasks die — measure it in your own logs.
  3. Distrust round numbers. When a vendor cites a real-world agent score, check the scoring rule before you believe it, exactly as you would a self-reported launch benchmark. All-or-nothing 32% and partial-credit 32% are different planets.

The point of a benchmark this unforgiving isn't to be discouraging. It's that a number the labs can't easily flatter is the only kind worth planning against. 32.5% is that number. Read the gap, scope the job, and ship the narrow thing first.