How to Test an LLM Feature Before You Ship It (a Minimal Eval Harness You Can Build in an Afternoon)

You wouldn't merge a change to your billing code without a test proving it charges the right amount. Yet the standard way to ship an LLM feature is: tweak the prompt, eyeball three examples in a playground, and deploy. That's not testing — it's a vibe check. And the reason it feels acceptable is that LLM output looks fine most of the time, which is exactly the property that lets a real regression hide in the 5% of cases you didn't paste in.

The fix is an eval: a small, automated test suite for your LLM feature that you run before every deploy and gate CI on. This is a start-to-finish guide to building the smallest one that's actually useful — you'll have it running this afternoon. We'll do it twice: once with promptfoo (a YAML file, zero code), and once as a framework-free Python harness you fully own.

What an eval actually is#

Four pieces, no more:

1. A dataset — a list of inputs, each paired with what a good answer looks like. Start with ~20 real cases, not 500 synthetic ones.
2. A task function — the thing under test: your prompt + model + tools. This is your product code.
3. Graders — functions that score each output.
4. A score — aggregate pass/fail into a number you can gate on (e.g. "90% must pass").

The whole craft is in the graders, and they come in two families you must not confuse.

"Does it work" vs "is it good"

Deterministic assertions are pure code with a checkable ground truth: output is valid JSON, contains the order number, matches a regex, is one of a fixed set of labels, came back under a latency budget. They're free, instant, and 100% reproducible. These are your regression tripwires — a passing assertion should almost never flip to failing unless something genuinely broke.

LLM-as-judge checks use a second, cheaper model to score open-ended quality against a rubric: is the answer faithful to the provided context? Is the tone professional? Did it avoid revealing the system prompt? Use these only where no single correct string exists.

The rule: assertions answer "does it work," rubric checks answer "is it good." Run the cheap deterministic checks first, and spend a judge call only on the dimensions code genuinely can't judge. Report the two separately so a fuzzy quality wobble never masks a hard structural break.

Path A — promptfoo (no code)#

promptfoo is an open-source CLI that runs a matrix of prompts × providers × test cases and gives you a pass/fail view. No install needed:

npx promptfoo@latest init
# edit promptfooconfig.yaml, then:
npx promptfoo@latest eval
npx promptfoo@latest view      # optional web UI

The config is three top-level keys — prompts, providers, tests — and every test carries an assert array (singular assert, a common gotcha). A minimal, real config:

# promptfooconfig.yaml
description: "Support-reply feature eval"

prompts:
  - "You are a support agent. Answer the customer in {{language}}. Question: {{input}}"

providers:
  - anthropic:messages:claude-sonnet-5

# Grade all llm-rubric checks with a cheap model
defaultTest:
  options:
    provider: anthropic:messages:claude-haiku-4-5

tests:
  - vars:
      language: French
      input: "Where is my order?"
    assert:
      - type: icontains          # deterministic: mentions "order" in French
        value: "commande"
      - type: llm-rubric         # model-graded: quality
        value: "Replies in French, is polite, and asks for an order number."

  - vars:
      language: English
      input: "Give me my account as JSON with fields name and email."
    assert:
      - type: is-json            # deterministic: structurally valid
      - type: llm-rubric
        value: "Returns exactly name and email, with no extra commentary."

npx promptfoo eval exits non-zero when assertions fail, so in CI it's a one-line gate. The provider string is anthropic:messages:<model-id> and expects ANTHROPIC_API_KEY in the environment (swap in openai: and OPENAI_API_KEY if you're on GPT). Note the split: icontains and is-json are your free tripwires; llm-rubric is where the cheap judge model earns its keep.

Path B — framework-free Python (~60 lines you own)#

Sometimes you want the harness inside your own repo, in your own language, with no new dependency to learn. Here's the whole thing with the official anthropic SDK (pip install anthropic, ANTHROPIC_API_KEY set). The task model is the feature under test; the judge is the cheap tier.

"""Tiny LLM eval harness — no framework. pip install anthropic; set ANTHROPIC_API_KEY."""
import json
from dataclasses import dataclass, field
from typing import Callable

import anthropic

client = anthropic.Anthropic()
TASK_MODEL = "claude-sonnet-5"     # the feature under test
JUDGE_MODEL = "claude-haiku-4-5"   # cheap grader for LLM-as-judge


# ---- the feature under test -------------------------------------------------
def task(user_input: str) -> str:
    resp = client.messages.create(
        model=TASK_MODEL, max_tokens=1024,
        system="You are a support agent. Reply concisely and, when asked for "
               "structured data, return only valid JSON.",
        messages=[{"role": "user", "content": user_input}],
    )
    return next((b.text for b in resp.content if b.type == "text"), "")


# ---- graders: each returns (passed, detail) ---------------------------------
Grader = Callable[[str], tuple[bool, str]]

def contains(substr: str) -> Grader:
    return lambda out: (substr.lower() in out.lower(), f"contains {substr!r}")

def is_json(out: str) -> tuple[bool, str]:
    try:
        json.loads(out); return True, "valid JSON"
    except ValueError as e:
        return False, f"invalid JSON: {e}"

def llm_judge(rubric: str) -> Grader:
    """Cheap model, forced single-word PASS/FAIL against one criterion."""
    def g(out: str) -> tuple[bool, str]:
        prompt = (f"Grade this response against the rubric.\nRUBRIC: {rubric}\n\n"
                  f"RESPONSE:\n{out}\n\nReply PASS or FAIL on the first line, then a reason.")
        r = client.messages.create(model=JUDGE_MODEL, max_tokens=100,
                                   messages=[{"role": "user", "content": prompt}])
        verdict = next((b.text for b in r.content if b.type == "text"), "").strip()
        return verdict.upper().startswith("PASS"), verdict.replace("\n", " ")[:80]
    return g


# ---- test cases: your living spec -------------------------------------------
@dataclass
class Case:
    name: str
    input: str
    graders: list[Grader] = field(default_factory=list)

CASES = [
    Case("refund_tone", "I want a refund, this is ridiculous!",
         [llm_judge("Stays calm and professional; does not promise a refund outright.")]),
    Case("json_shape", "Give my account as JSON with fields name and email.",
         [is_json, llm_judge("Contains only name and email fields, no prose.")]),
    Case("mentions_order", "Where is my order?", [contains("order")]),
]


# ---- run + report + gate ----------------------------------------------------
def main() -> None:
    rows, passed, total = [], 0, 0
    for case in CASES:
        out = task(case.input)
        for grader in case.graders:
            ok, detail = grader(out)
            total += 1; passed += ok
            rows.append((case.name, ok, detail))

    print(f"{'CASE':<18}{'RESULT':<8}DETAIL\n" + "-" * 60)
    for name, ok, detail in rows:
        print(f"{name:<18}{'PASS' if ok else 'FAIL':<8}{detail}")
    score = passed / total if total else 0.0
    print("-" * 60 + f"\nSCORE: {passed}/{total} = {score:.0%}")

    if score < 0.9:                       # the CI gate
        raise SystemExit(f"Eval below threshold ({score:.0%} < 90%)")


if __name__ == "__main__":
    main()

Run it, and the last four lines are the whole point: a sub-threshold score raises SystemExit, which returns a non-zero exit code, which fails your build. That single line turns a script into a deploy gate.

Make it a habit, not a heroic afternoon#

The harness is 20% of the value. The other 80% is the discipline around it:

• Keep evals in version control, in an evals/ directory next to the code they test. Your dataset and rubrics are part of the product spec — review them like code.
• Run them in CI on every PR that touches a prompt, model, or the logic around it. Both paths above exit non-zero on failure, so it's a one-line step in GitHub Actions.
• Gate on a threshold, not perfection. Require all deterministic assertions to pass (those are structural — a regression there is a real break), and set the rubric score to a threshold like 90%, since the judge is noisy.
• Start with ~20 cases covering real failure modes — the malformed input, the injection attempt, the tone complaint — and add a case every time a bug reaches production. The dataset grows into a regression net that gets stronger with every incident.
• Track the score over time, not just this run. A slow drift down after a model or prompt change is precisely the failure evals exist to catch — and the thing a vibe check will never show you.

When you outgrow the afternoon version, the graduation path is Braintrust, Langfuse, or OpenAI Evals for dashboards, team datasets, and long-term tracking. But you don't need any of them to start. You need twenty real cases and the honesty to run them before you ship.