The short version: most teams instrument their agent, admire a dashboard, and never turn a single production failure into a test — so the same bug ships twice. The high-leverage move isn't a bigger dashboard; it's a closed loop you can build in an afternoon in Langfuse. Three steps: (1) filter your low-confidence production traces with langfuse.api.trace.list(); (2) route the ambiguous ones to a human annotation queue with one POST so a domain expert labels pass/fail and writes the corrected output; (3) promote those labeled failures into a versioned regression dataset with create_dataset_item(source_trace_id=...) that gates CI. The queue, the scores, and the dataset share one data model, so there's no second tool to buy. Below is the runnable version, verified against the Langfuse Python SDK v4 and public API.

Why a loop beats a dashboard#

Observability tells you that the agent failed. It does nothing to stop the failure recurring. The artifact that stops recurrence is a regression case: a real input, the output the agent should have produced, and a check that runs on every deploy. As we've argued, your eval set is a precipitate of error analysis — you distill it from failures you actually observed, not ones you imagined. The problem is the plumbing: getting a flagged trace in front of the one human who can label it correctly, and getting that label back into a dataset without a pile of glue code.

Langfuse already has the three primitives this needs — traces, annotation queues, and datasets — and they all speak the same score model. So the loop is short.

Step 1: find the traces worth a human's time#

Don't ship everything to a human; human attention is the scarce resource. Flag traces at runtime — when a guardrail trips, a tool returns empty, or an online judge scores low — by tagging them. Then the pull is a one-liner. The current SDK is v4 (released March 2026):

from langfuse import Langfuse
from datetime import datetime, timedelta

langfuse = Langfuse()  # reads LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / LANGFUSE_HOST

# Traces you tagged "needs-review" at runtime, from the last week.
traces = langfuse.api.trace.list(
    tags=["needs-review"],
    from_timestamp=datetime.now() - timedelta(days=7),
    limit=50,
)

for t in traces.data:
    print(t.id, t.name)

If you'd rather flag traces automatically, attach a verdict programmatically and filter on it later. create_score writes to the exact same score model a human reviewer uses:

langfuse.create_score(
    trace_id=t.id,
    name="auto_judge",
    value=0,                 # 0 = the judge thinks this failed
    data_type="BOOLEAN",
    comment="tool call returned an empty result",
)

Langfuse can also run an LLM-as-a-judge on individual observations — a single tool call or retrieval, not just the whole trace — and write that score at ingest, which is the cleanest way to auto-populate your needs-review set without writing the judge loop yourself.

Step 2: route them to an annotation queue (one POST)#

Create the queue once in the UI (Annotations → New queue), give it a score config — say a boolean human_verdict plus a categorical failure_mode — and copy its ID. From then on, adding items is a single authenticated REST call. The endpoint takes the object's ID, its type (TRACE, OBSERVATION, or SESSION), and an initial status:

import os, requests

QUEUE_ID = "your-queue-id"
host = os.environ["LANGFUSE_HOST"]  # e.g. https://cloud.langfuse.com
auth = (os.environ["LANGFUSE_PUBLIC_KEY"], os.environ["LANGFUSE_SECRET_KEY"])

for t in traces.data:
    r = requests.post(
        f"{host}/api/public/annotation-queues/{QUEUE_ID}/items",
        auth=auth,
        json={"objectId": t.id, "objectType": "TRACE", "status": "PENDING"},
    )
    r.raise_for_status()

Or, if you're scripting from the shell:

curl -s -X POST \
  "$LANGFUSE_HOST/api/public/annotation-queues/$QUEUE_ID/items" \
  -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{"objectId":"'"$TRACE_ID"'","objectType":"TRACE","status":"PENDING"}'

Now your reviewer opens the queue and works through a clean list — the full trace on one side, the score config on the other. They produce two things that both land as scores on the trace: a verdict (human_verdict: pass/fail, or a failure_mode label) and, in the comment or a text score, the corrected output — what the agent should have said. The corrected output is the part that matters. A verdict alone tells you the trace is bad; the correction turns it into a test.

One discipline worth borrowing from the eval literature: let a single domain expert own the bar. A "benevolent dictator" labeler — a lawyer for a legal tool, a clinician for a health bot — kills the annotation deadlock that stalls review projects. Add multiple annotators only at scale, and measure their agreement when you do.

Step 3: promote the labeled failures to a regression dataset#

Once a batch is reviewed, pull the traces a human marked as failures and write each one into a dataset. create_dataset_item upserts the item and — this is the good part — source_trace_id keeps a hard link back to the trace it came from, so every regression case is traceable to the real incident that spawned it:

langfuse.create_dataset(name="agent-regressions")

for t in reviewed_failures:              # traces where human_verdict == fail
    langfuse.create_dataset_item(
        dataset_name="agent-regressions",
        input=t.input,                   # the real production input
        expected_output=corrected[t.id], # the reviewer's corrected output
        metadata={"failure_mode": mode[t.id], "trace_id": t.id},
        source_trace_id=t.id,            # provenance back to the incident
    )

That's the whole loop: a flagged production trace is now a labeled, versioned, provenance-carrying regression case. You didn't invent the input, and you didn't guess the expected output — a human who knows the domain corrected it.

Wire it into CI and keep it alive#

The dataset is only worth building if it runs. Pull it back down with get_dataset and diff your agent's output against each expected output as part of your test suite:

dataset = langfuse.get_dataset("agent-regressions")

for item in dataset.items:
    output = run_your_agent(item.input)
    assert passes(output, item.expected_output), f"regressed on {item.metadata['trace_id']}"

Fail the build on a regression and the same bug can't ship twice. Two cautions we cover elsewhere: don't let a stochastic agent turn this into a flaky gate — score on behavior a crisp check can capture, and run enough samples to separate signal from noise — and treat the set as living, not frozen. Every week's new failures feed a new batch through the same queue. See our CI eval-gate walkthrough for the full harness.

The triage that makes this cheap#

The reason this scales is that humans only ever see the residue. Run code assertions on every trace (schema, format, empty tool results — free and deterministic), an LLM judge on the sampled subset where semantic judgment matters (a few cents each), and route only the disagreements and low-confidence cases to the human queue. Anthropic puts the useful starting point at just 20–50 real failures, because early in a product's life each fix has a large effect size — fifty real, corrected cases beat five hundred synthetic prompts. You are not trying to review everything. You are trying to catch the handful of failures worth encoding forever, get them corrected by the one person who can, and never see them again.