The slowest part of building good evals has always been writing the rubric. Deciding what "correct" means for your agent — enumerating the dimensions, weighting them, arguing with a domain expert about edge cases — is expert-hungry, tedious work, and it is the work most teams skip. So it was only a matter of time before someone offered to do it for you.

That someone is now Microsoft. Foundry's Rubric evaluator, in public preview since the Build 2026 wave, takes a description of what your agent is supposed to do and auto-generates a weighted grading rubric — spreading a score across task success, tone, safety, cost, and latency — then grades each run against it. Google's Vertex AI ships the same idea as per-prompt Adaptive Rubrics. Microsoft's recommended action is not subtle: generate a rubric and "replace your static benchmarks."

Take the offer at face value for a second, because the convenience is genuine. Then look at the shape of the machine you just assembled.

Count the LLM calls#

There is a first model call that reads your task and writes the criteria. There is a second model call that reads an output and scores it against those criteria. And if the thing being graded is an LLM agent — which is the entire point — then there is a third model call upstream that produced the output in the first place.

Three model calls in a row, and not one of them touches the outside world.

A hand-written rubric is a human judgment frozen into a checklist. An auto-generated one is the model's judgment about what the model should be judged on. The anchor is gone.

This is the part the demo skips. A rubric is supposed to be the fixed point — the thing that doesn't move when the model does, the outside standard you hold the output against. Auto-generation removes exactly that property. The criteria are now downstream of the same intelligence that writes the answers, drawn from the same well of assumptions about what a good answer looks like.

Coherence is not correctness#

Here is the specific failure, stated plainly: a rubric derived from the task specification measures whether the output matches the kind of thing the prompt asked for. It does not measure whether the answer is true in the world. Most of the time those agree, which is why the technique demos well. They come apart precisely on the cases that matter — the confident, fluent, well-structured answer that is also wrong.

The research literature has been circling this for two years, and three findings line up on it. The authors of G-Eval — one of the first LLM-as-judge methods — noted that their evaluator gave GPT-3.5 summaries higher scores than human-written ones even when human raters preferred the humans, and proposed a mechanism: the model "could share the same concept of evaluation criteria during generation and evaluation." It grades toward its own style because it wrote the style guide. Later work on self-preference bias made it sharper: an LLM judge scores its own outputs higher than humans do, and the effect grows in proportion to how well the model can recognize its own writing.

And the EvalGen paper — pointedly titled Who Validates the Validators? — found the load-bearing problem for auto-rubrics specifically: evaluation criteria are output-dependent. Users can't fully define what to grade until they've looked at real outputs and graded some by hand; the criteria drift as they do. Which means a rubric generated before any human has looked at the outputs is not just incomplete, it is measuring a target no one has validated. Their conclusion is the one Foundry's marketing elides: LLM-generated evaluators inherit the problems of the LLMs they evaluate, and need a human to check them.

The tell is in Microsoft's own numbers#

Microsoft reports case-level agreement between its auto-rubric and trusted signals as an ROC AUC — and the numbers are quietly honest about the boundary. On a JSON-editing task, where "correct" is checkable and close to ground truth, agreement is about 0.972. On a fuzzier customer-support benchmark, it drops to around 0.794. The technique is strongest exactly where you least need it — where an outside oracle already exists — and weakest where you were hoping it would save you, on the subjective tasks that have no oracle at all. That is not a knock on the engineering; it is the signature of what the method can and cannot do.

None of this makes auto-rubrics useless. It makes them a draft, not a gate. As a way to bootstrap coverage across a hundred dimensions you'd never hand-write, to catch the obvious formatting and safety failures, to get something running on day one — they are a real accelerant. Microsoft's own guidance says as much between the lines: review the generated rubric carefully, and run it against known-good and known-bad cases before you rely on it.

The line to hold is about what you ship against. The rubric that blocks a release — your regression bar, your acceptance gate — needs an anchor outside the model: a small set of human-labeled cases, scored on precision and recall rather than raw agreement, that the auto-rubric has to earn its trust against. This is the same discipline that keeps a judge from silently drifting and the same reason an LLM judge is only as good as its calibration against humans. Skip it, and you have built something that feels like measurement and functions like a mirror: the model writes the answer, writes the test, and marks its own paper — and every grade it returns is a grade it was always going to give itself.