If you read one line: Attach a score to every trace, create a Langfuse Monitor that watches that metric over a time window with a warning and an alert threshold, and link it to a Slack, GitHub Actions, or webhook automation — that is the whole loop that turns "quality quietly regressed" into a message that pages you.

You already know why agents fail in production: the model does something slightly worse, cost creeps, latency drifts, and nobody notices until a user complains. A dashboard only helps if someone is looking at it. Monitors, which Langfuse shipped in June 2026 and extended to boolean scores in July, close that gap. Here is the minimal setup.

1. Decide what "bad" looks like before you wire an alert#

Pick two or three signals with a number you can defend, not everything you can measure. The compare table above is a starting menu: a cost ceiling per trace, a p95 latency SLA, an error count, and — the reason this got interesting — a rate of failed policy checks or hallucinations. Write the threshold down first. If you cannot name the number that means "wake me up," you are not ready to create the monitor.

What it means for you: one good alert beats ten noisy ones. A solo founder wants to be paged when revenue or trust is at risk, not when p95 twitched for ninety seconds.

2. Attach a score to every trace#

A monitor can only watch data you send. Cost and latency come free with tracing — if you have not instrumented yet, start with wiring Langfuse into your agent over OpenTelemetry. Quality signals you attach yourself as scores.

For quality you almost always want a boolean score: did this response pass the policy check, or did the hallucination detector flag it? Ingest it with the Python SDK:

from langfuse import get_client

langfuse = get_client()

# 1 = passed the policy check, 0 = failed
langfuse.create_score(
    name="policy_check_passed",
    value=1,                 # 0 or 1
    trace_id="trace_id_here",
    data_type="BOOLEAN",     # required; a bare number would be inferred as NUMERIC
    comment="No PII in the response",
)

If you are computing the check inside an instrumented span, score the current trace directly instead of tracking the ID:

with langfuse.start_as_current_observation(as_type="span", name="answer-user"):
    # ... agent logic runs, you run your check ...
    langfuse.score_current_trace(
        name="hallucination_detected",
        value=0,             # 0 or 1
        data_type="BOOLEAN",
    )

The trick that makes boolean scores worth the trouble: the average of a boolean score is the share of results that are true. So an average of policy_check_passed is your pass rate, and an average of hallucination_detected is your hallucination rate — both directly alertable.

A boolean score you never read is dead weight. A boolean score behind a monitor is an on-call engineer that never sleeps.

3. Create a Monitor#

In your project, open Monitors and click New Monitor. You configure three things:

Name it, save, and it goes ACTIVE and schedules its first evaluation immediately. Langfuse's own comparison of Monitors against Braintrust and Phoenix is worth a read if you are weighing platforms.

4. Route the alert (Slack / GitHub Actions / webhook)#

Notifications live in Automations, separate from monitors so you can reuse them. Click Create Automation, choose event source Monitor, then an action type:

The webhook payload carries the essentials — severity, a human message.title and body, and a permalink straight to the monitor:

{
  "type": "monitor-alert",
  "payload": {
    "severity": "ALERT",
    "message": {
      "title": "avg latency crossed alert threshold",
      "body": "avg latency is 1234 ms (threshold: 1000 ms) over the last 1 hour"
    },
    "permalink": "https://cloud.langfuse.com/project/.../monitors/..."
  }
}

Verify the signature exactly as you would for prompt webhooks — the HMAC scheme is shared. Then, back in the monitor editor, select the automation in the Automations panel and save to link them.

What it means for you: the monitor decides when; the automation decides where. One monitor can fire all three channels at once, so a spike can ping Slack and open a GitHub issue in the same beat.

5. Tune thresholds so you get paged, not spammed#

Monitors notify on severity transitions, not every check: OK to WARNING or ALERT fires once, and recovery back to OK fires once. With Renotify off (the default) a metric parked at the threshold will not re-page you. Use the warning threshold as a soft heads-up and reserve ALERT for the number that justifies interrupting dinner. If your endpoint flaps, note the safety valve: after 5 consecutive delivery failures Langfuse disables that automation's trigger until you re-enable it.

Because Langfuse is MIT-licensed and now inside ClickHouse, none of this is a paid add-on gate — monitors ship on the free Hobby tier (two of them) and on self-hosted v4.

Ship it#

The minimum before you sleep: one boolean quality score on every trace, one monitor on its average with an ALERT threshold and a 1-day window, and one Slack automation linked to it. Add a cost-per-trace monitor and a p95-latency monitor next. That is three monitors and one Slack channel standing between you and a silent regression — set it up once, then stop watching the dashboard.