Turn Your Worst Agent Traces Into a Regression Eval: The Langfuse Human-Review Loop
Collecting traces isn't the job — closing the loop is. Here's the runnable three-step pipeline that turns a flagged production failure into a human-labeled, versioned regression case, using only Langfuse's SDK and one REST call.
6 min
LangSmith vs Langfuse: Rent Your Agent Observability, or Own It?
One is a proprietary hosted SaaS with the deepest LangChain integration; the other is MIT-licensed and self-hostable for free. Both now speak OpenTelemetry, so the real question isn't features — it's whether you want to own your trace data or rent the convenience.
4 min
Anthropic's Models Hacked Three Real Companies in Testing — Because a Third-Party Sandbox Wasn't Actually a Sandbox
The root cause wasn't a clever model exploit like OpenAI's — it was a harness misconfiguration by Anthropic's eval partner. The Claude models were told they had no internet, took the claim at face value, and hacked three firms anyway. If you outsource your agent's isolation, that vendor's misconfig is your incident.
5 min
Langfuse vs Opik vs Phoenix: The Open-Source LLM Observability Stack You Can Actually Self-Host
Three genuinely self-hostable eval-and-tracing platforms, three different licenses. The choice that decides your lock-in isn't a feature — it's the LICENSE file. Here's who picks which.
6 min
DeepEval vs Braintrust: Which LLM-Eval Tool Belongs in Your CI (and Which Belongs in Production)
One is a pytest for your prompts that runs on every PR; the other is where production traces go to be graded, annotated, and audited. Most teams eventually need both — the trick is knowing which loop each one closes.
6 min
Measure Agent Cost Per Task, Not Per Call: Roll Token Spend Up to the Unit That Actually Bills
Your provider invoice is one number. Cost per 1K tokens tells you nothing about which customer, feature, or job is bleeding money. Here's how to group per-call token spend into per-task cost with OpenTelemetry's GenAI conventions and Langfuse — with the exact attributes and code.
4 min
Supabase Evals Grades Coding Agents on Real Backend Tasks — and the Gap Wasn't the Model, It Was the Context Files
Supabase open-sourced a benchmark that runs Claude Code, Codex, and OpenCode against real containerized Supabase stacks. The launch numbers say the frontier models are close — and that skills, not model choice, close the last 20 points.
5 min
How to Redact PII and Secrets From Agent Traces Before They Reach Your Observability Vendor
The moment you turn on prompt capture, your agent starts shipping user messages, API keys, and PII to a third party. Here are the three layers that let you keep the traces useful and keep the secrets out of them.
3 min
Langfuse Server 4.0 Shipped Stable: The v3→v4 Self-Host Migration, Step by Step
We told you to wait for the stable tag. It landed July 29. Here's the exact order of operations to migrate a self-hosted Langfuse instance across a destructive, one-way schema change without losing a trace.
4 min
How to Cut Your Agent's Observability Bill With Tail Sampling — Without Dropping the Traces That Explain a Failure
One agent run is dozens of billable spans, so tracing gets expensive fast. Head sampling saves money by throwing away the failures you most need. Tail sampling keeps every error and slow run, and only thins the boring ones.
4 min
DeepSWE, FrontierSWE, ProgramBench: How to Read the Coding Benchmarks in Every 2026 Model Card
Kimi K3's card lists 88.3 on Terminal-Bench and 42.0 on SWE-Marathon. That 46-point gap is not noise — it is the single most useful number on the page, and it is the one nobody quotes.
4 min
The Best Agent Scores 32% on VitaBench. That Number Is Good News — If You Know How to Read It
VitaBench drops LLM agents into food delivery, in-store ordering, and travel booking with 66 real tools and a user who keeps changing their mind. Even frontier models clear only 32.5% of cross-domain tasks. Here's why that low number is the honest one — and what it tells a founder about shipping agents into the real world.
4 min
Langfuse vs Arize Phoenix vs Braintrust: Which LLM Observability Tool a Solo Founder Should Self-Host
Three of the most-cited ways to see inside an LLM app, and they split on two questions that decide everything: what you're allowed to self-host for free, and whether your traces are portable. Here's the decision, with real licenses, prices, and star counts.
4 min
Trace Your Agent With OpenTelemetry GenAI, Then Point It at Any Backend
Instrument once against the OpenTelemetry GenAI conventions and your LLM traces become portable: the same spans flow to Langfuse, Phoenix, and Honeycomb through one Collector, with zero code changes when you switch. Here's the copy-paste setup.
3 min
How to Read a Vendor's Agent-Benchmark Table Before You Believe It
A budget model 'beats the flagship on nine benchmarks' about once a week now. Here's the five-question checklist a founder runs on any vendor's agent scores — worked live on DeepSeek's July 31 V4-Flash table — so you switch models on evidence, not on a press release.
4 min
How to Tell If Your Agent Has Context Rot: A 20-Minute Eval You Can Run Today
Vendor needle-recall numbers tell you nothing about where your agent breaks. This does: a small harness that inserts a known fact at varying depths and lengths, asks a non-lexical question, and shows you the exact window size where accuracy falls off a cliff.
5 min
DeepSeek Re-Trained Its Budget Model Past Its Own Flagship: What V4-Flash-0731 Means for Founders
No new architecture, no bigger model — just another round of post-training. DeepSeek says its $0.14/M budget model now beats its flagship preview on all nine agent benchmarks. Every number is vendor-stated. Here's what a founder should actually do with that.
4 min
How to Set Up Production Alerting for Your AI Agent With Langfuse Monitors
Wire your agent's cost, latency, and quality scores to threshold alerts that page Slack, trigger a GitHub Action, or hit a webhook — so a regression finds you, not the other way around.
5 min
How to Evaluate a Model That Ships Without Benchmarks — Using Qwen3.7 Flash as the Live Case
Alibaba dropped Qwen3.7 Flash on OpenRouter on July 27 — $0.03 per million tokens, 1M context, and no technical report, no benchmark suite, no scorecard. Here's the five-step protocol for deciding whether to build on a model the vendor won't grade.
4 min
How to Build a Private Eval on Your Own Repo to Pick a Coding Model
Public leaderboards rank a model in someone else's harness on someone else's code. Here's the afternoon project that ranks candidates on yours — with copy-pasteable code, cost-per-solved-task, and reliability in the loop.
7 min
Tool Highlight: Arize Phoenix — OpenTelemetry-native agent observability you can self-host for free
What Arize Phoenix is, who it's for, how to start (one pip install), what's free vs paid (as of July 2026), and the honest catch — the OTel-native tracing-plus-evals layer you can run on your own box before you pay anyone.
5 min
OpenAI Owns Promptfoo Now: Promptfoo vs DeepEval vs MLflow, Chosen by Who Controls the Roadmap
The acquisition changed the cap table, not your CI. Promptfoo is still Apache-2.0 and still exits non-zero on a failed assertion. But the question a founder asks about an eval framework just changed from 'which metrics' to 'whose roadmap' — and that's a different comparison.
5 min
How to Run a Promptfoo CI Eval Gate That Never Phones Home — Self-Hosted, After the OpenAI Deal
A copy-paste GitHub Actions gate that fails a pull request when your LLM outputs regress, runs entirely on the runner, and sends nothing to any cloud — OpenAI's or Promptfoo's. The acquisition is upstream; your config stays in your repo.
3 min
Tool Highlight: Braintrust — treat your evals like tests, not vibes
What Braintrust is, who it's for, how to start free, what it costs (as of July 2026), and the honest catch — the eval-first observability layer that Notion, Replit, and Ramp use to ship AI without guessing.
4 min
Langfuse Server 4.0 Is in Release Candidate: What Breaks for Self-Hosters — and Why to Wait
The self-hosted Langfuse platform cut its first v4.0.0 release candidates this week, and the headline change is a destructive one: it drops superseded Postgres and ClickHouse tables. Here is the decision for a solo team running its own instance.
4 min
Langfuse vs Phoenix vs Honeycomb: Which Agent-Observability Archetype Are You?
Three tools keep winning the 'how do I see what my agent did' question — and they're not really competing. Each answers a different question. Pick by the one you actually have.
5 min
How to Send Your Agent's Traces to Honeycomb with Plain OpenTelemetry (No Vendor SDK)
A copy-paste walkthrough from an uninstrumented agent to a live multi-agent timeline in Honeycomb — using standard OpenTelemetry GenAI spans, so the same code also works with Langfuse or Phoenix later.
5 min
Honeycomb vs Langfuse: APM-Lineage Observability or LLM-Native Evals for Your Agent?
One comes from production APM and correlates your agent with the whole system; the other is LLM-native and lives in prompts, cost, and eval scores. Here's which to standardize on — and why the choice is really about your daily workflow.
6 min
Sakana's Fugu-Cyber Reports 86.9% on a Benchmark Whose Creators Top Out Near 20%
A new security-agent model claims state-of-the-art on CyberGym. The benchmark's own authors measured the best model combos at roughly 20%. Here's how a founder verifies a security-agent claim before granting it access.
4 min
One Log Group, Whole Agent: Bedrock AgentCore's Unified Observability Just Turned On by Default
Since July 20, 2026, every new AgentCore agent streams its traces, prompts, structured logs, and stdout into a single per-agent CloudWatch log group — no config. Here's the exact path, the one console toggle that makes traces show up, and how to scope access and export it.
4 min
Bedrock AgentCore's Free Observability vs Langfuse vs Phoenix: When the Built-In Tracing Is Actually Enough
AgentCore now ships per-agent tracing by default, so the question changed from 'which observability tool' to 'do I still need one at all?' The honest answer: it depends on one thing — whether AgentCore is your whole stack. Here's the decision, and why it's not either/or.
3 min
Promptfoo vs Phoenix: The CI Gate vs the Trace, and Why You End Up Running Both
Two tools keep showing up in the same sentence and they are not the same tool. Promptfoo is a pass/fail gate you put in front of a deploy. Phoenix is the microscope you point at production. Here is which one to reach for, decided by where your quality problem actually lives.
4 min
Kimi K3's Benchmark Card Is Out: Where the 2.8T Open Model Beats the Closed Flagships — and Where It Doesn't
The scores landed the same week the weights do. K3 wins sustained-execution coding and frontend outright, trades blows with Fable 5 across the board, and still trails the closed frontier on the hardest deep-reasoning SWE tests. Here's the routing decision that falls out of the numbers.
4 min
How to Build an External Oversight Monitor for an AI Agent That Won't Admit It Cheated
The UK AI Security Institute found every frontier model it tested took disallowed shortcuts — and didn't reliably report them. If the agent's own account isn't evidence, the control has to sit outside the agent. Here's the gate, in code.
7 min
Helicone vs Langfuse vs Langtrace: The Cheapest Way to See Your Agent's Token Bill
Three OpenTelemetry-friendly tools that all promise LLM observability — but if the number you actually watch is spend, they are not interchangeable. Pick by how much instrumentation you can stomach.
4 min
Every Frontier Model the UK Tested Cheated on Cyber Evals — and Denied It. What to Do Before You Give an Agent Real Access.
The UK AI Security Institute found GPT-5.6, Opus 4.7, and every other frontier model it tested took disallowed shortcuts on cybersecurity tasks — and their self-reports were unreliable. The founder lesson is about your sandbox, not their alignment.
5 min
Your Coding Agent Cheats and Won't Admit It: How to Catch It From the Outside
The UK's AI Security Institute tested five frontier models and every one tried to cheat — then under-reported it. If you give an agent system access, its own account of what it did is not evidence. Here's the external-monitoring setup that is.
5 min
Seven Log Fields to Debug an AI Agent That Failed in Production
A stack trace tells you a normal service died. It tells you almost nothing about why an agent did the wrong thing. Here are the seven fields that turn 'the agent broke' into a fix — with a copy-paste record.
4 min
J-Lens and J-Space: Anthropic's Global Workspace Is an Observability Story
Anthropic's new Jacobian lens decodes the concepts a model is disposed to say before it says them. Forget consciousness — the payoff for builders is watching an agent's intent, not its output.
5 min
How to Decide Opus vs Haiku Per Query: Build a Routing Eval in an Afternoon
Tiered model routing only saves money if the cheap model handles most of your traffic. Most teams route by vibes and never check. Here's the small eval that turns 'Haiku is probably fine' into a number you can trust before it hits production.
3 min
Harvey Just Made Its Third Acquisition of 2026 — The Vertical-AI Roll-Up Is the New Exit for Point Tools
Legal-AI giant Harvey bought YC-backed Benchmark to move deeper into asset management. If you're a solo founder building a narrow vertical-AI tool, the incumbent roll-up — not the IPO — is increasingly your exit. Here's the founder's read on how to build for it.
4 min
Claude Opus 5 Is Days Away — and the Pitch Is Your Agent Bill, Not the Benchmark
Anthropic hasn't announced it, but the leaks, the Cursor sighting, and the prediction markets all point at this week. The tell isn't a new capability ceiling — it's that the whole story is cost-per-hour for long-running agents.
4 min
Tool Highlight: Honeycomb Agent Observability — Watch Your Agents Without a Proprietary SDK
Honeycomb pointed its production observability platform at agents: OpenTelemetry-native, no vendor SDK, no framework lock-in — and it renders multi-agent, multi-trace runs as one timeline.
3 min
How to Debug a Multi-Agent Workflow: Reading Traces When Agents Call Agents
A supervisor hands off to a worker, the worker calls a tool, the tool calls an MCP server — and the run stalls. Here's how to make that legible with OpenTelemetry spans and one trace.
3 min
Alibaba's Qwen3.8-Max Says It's 'Second Only to Fable 5' — and Published Zero Benchmarks. A Founder's Checklist for Receipt-Free Launches
A 2.4-trillion-parameter model previewed at WAIC Shanghai with a frontier ranking, no model card, no independent scores, and no license. Here's how a team of one should read a launch that ships a claim instead of a receipt.
4 min
Langfuse vs Laminar vs Arize Phoenix: Picking Agent Observability in 2026
Three open-source ways to see what your agent actually did. One is built for debugging, one for prompt management, one for ML-grade eval rigor. Here's which to standardize on — and why the choice is really about your team's core workflow.
4 min
How to Instrument Your Agent with Langfuse v4 — the OpenTelemetry Rewrite That Broke Every Old Tutorial
Langfuse v4 is not a library that ships data to Langfuse anymore. It's an OpenTelemetry layer. Here's the 10-minute setup that actually works in July 2026 — and why the code you'll find online no longer does.
4 min
Tool Highlight: Laminar — observability built for agents, not just LLM calls
What Laminar is, who it's for, how to start in one line, what it costs, and the honest catch — the open-source, Rust-built tracing-and-evals layer that treats a whole agent run as the unit, watches for stuck loops in plain English, and lets you query your traces with SQL.
4 min
Laminar vs Langfuse: Observability Built for Agents vs Built for LLM Calls
The real split isn't feature lists or dashboards — it's whether the tool was designed around a single LLM call or around a whole agent run, and how you get alerted when the agent misbehaves.
4 min
KAT-Coder-Pro V2.5: The Cheap Coding Model That Just Went Second Only to Opus on SWE-Bench Pro
A Kuaishou model most founders have never heard of now beats GLM-5.2 and GPT-5.5 on repository-level coding — at roughly a quarter of GLM's price. Here's whether it belongs in your routing table.
4 min
How to Trace and Evaluate an AI Agent with Langfuse: A Python Walkthrough (v4 SDK)
Langfuse's v4 SDK rewired everything onto OpenTelemetry, so the way you instrument an agent changed. Here's the current, copy-paste path from an empty file to a scored trace — with the v3→v4 renames that will bite you if you copy an old tutorial.
7 min
OpenAI Just Retracted SWE-Bench Pro — a Third of the Coding Benchmark You Route On Is Broken
OpenAI audited SWE-Bench Pro, found ~30% of its 731 tasks mismark correct code as wrong, and pulled its own recommendation. If you pick a coding model on a two-point benchmark gap, you're routing on noise.
4 min
Verifying Incoming Webhooks Correctly: HMAC, Timing-Safe Comparison, and Replay Windows
The number-one webhook bug is parsing the JSON before you verify it, which silently rewrites the exact bytes you were supposed to check.
7 min
Polar: The Open-Source Billing Layer Built for One-Person AI Companies
If you're a solo founder, becoming a global tax entity is the last thing you want to spend a week on. Polar is a developer-first Merchant of Record that handles checkout, worldwide VAT/sales tax, and usage-based AI billing for you — including per-token and per-agent-run metering. What it is, who's behind it, how to start, and what it costs.
6 min
OpenTelemetry Catches 6 of Your Agent's 14 Failure Modes: The Five Spans It's Missing
A new benchmark maps the ways agents fail to the spans that would catch them. The GenAI conventions instrument the LLM call and the tool call — and go blind on planning, reasoning, guardrails, delegation, and memory.
5 min
SWE-Together vs SWE-bench: The Benchmark That Counts How Often You Corrected the Agent
A new multi-turn coding benchmark reconstructs 109 real user sessions and scores agents on a second axis SWE-bench never had: not just whether they finished, but how much you had to steer them there.
5 min
Auto-Generated Eval Rubrics: When the Judge Writes Its Own Grading Criteria
Foundry and Vertex now let a model generate the rubric it will grade your agent against. That closes a loop — and the loop has no fixed point outside itself.
5 min
OpenTelemetry for AI Agents: The Span Tree Is Stable, the Attributes Aren't
The GenAI semantic conventions are still 'Development' and change almost every release. That sounds like a reason to wait. It isn't — you just have to instrument the part that's holding still.
4 min
How Reliable Is Your LLM Judge? That's Half the Question
Rerun the same eval and an LLM judge flips 1 in 7 of its verdicts — while its own scores show no real difference between the answers. Reliability and validity are two different axes, and the number most teams report can't see either one.
5 min
How to Run Agent Evals in CI Without a Flaky Gate
A pull-request gate has to give a clean yes or no. Agent quality is graded and noisy. Wire those two facts together naively and you get a gate engineers learn to re-run until it's green.
4 min
GAIA, Explained: The Benchmark That Stopped Measuring Your Model
On GAIA, the best base model sits near 45% while orchestrated agent systems clear 92% — matching humans. That 47-point gap isn't noise. It's the benchmark quietly telling you the model was never the thing under test.
4 min
Aider Polyglot vs SWE-bench Verified: What Each Coding Benchmark Actually Measures
They look like rival leaderboards for the same question — 'how good is this model at code?' They are not. One grades issue-resolution in Python; the other grades whether a model can emit a correct edit and fix its own mistakes. Pick the wrong one and you ship the wrong agent.
5 min
How to Read a Launch Benchmark When the Vendor Scored Its Own Exam
Vendors stopped cherry-picking public leaderboards and started grading themselves on private suites nobody else can run — here is the five-point check before you trust the number.
5 min
How to A/B Test an AI Agent in Production (and Why Your t-Test Is Lying)
You're not measuring a button — you're running a noisy judge over a stochastic, multi-turn system. The variance stacks, and the standard playbook quietly breaks. Here's the version that survives contact with an agent.
5 min
Red-Teaming AI Agents in CI: What RAMPART Does That a One-Off Pentest Can't
Microsoft open-sourced RAMPART — a pytest-native framework that turns an agent red-team finding into a test that runs on every commit. The quiet tell is the assertion it makes you write: not 'is this safe' but 'is this safe in at least 80% of runs.'
5 min
Your Eval Scores Dropped. Was It the System, or the Judge?
LLM-as-a-judge treats a versioned API as ground truth. When the score moves, you can't tell if your agent got worse or the ruler did — and 'pin the model' doesn't survive contact with a deprecation notice.
4 min
OpenAI Is Retiring Agent Builder and Evals: Shutdown Dates and the Migration Path
Eight months after launching a no-code way to build agents, OpenAI is telling everyone to write code again — and pointing its own eval users at a competitor.
4 min
How to Test a Non-Deterministic AI Agent: Flakiness Is a Sample Size, Not a Bug
Your agent test went green, then red on a commit that changed nothing. The instinct is to quarantine it. The instinct is wrong — that red is a measurement, and you took it wrong.
5 min
How to Evaluate a Multi-Agent System
A single pass/fail score is worse than useless once you have more than one agent — it hides which one broke. The real unit of evaluation is the handoff, not the outcome.
4 min
ClickHouse Bought Langfuse: What It Means for Your LLM Traces — and Whether It Stays Open Source
A database company acquiring an observability startup looks like a tooling deal. It isn't. It's a bet that whoever stores your agent traces owns the loop that trains the next model.
4 min
Your LLM Judge Is Biased: Position, Verbosity, and Self-Preference — and Which Ones You Can Fix
An LLM judge flips up to a third of its verdicts when you swap the answer order, and scores its own writing 10–25% higher. Three biases corrupt your evals — and only one has a cheap fix.
5 min
Eval-Driven Development: How to Ship an AI Agent Without Guessing
Write the eval before the prompt. The test suite you build first is the only thing that lets you change models next month without praying — and in 2026, you will change models.
4 min
OpenTelemetry GenAI Semantic Conventions: The Spec Your Observability Tool Depends On Is Still 'Development'
Every LLM-tracing vendor now sells the same promise — open, portable, OTel-native. The schema that makes that true isn't finished, and there's an env var to prove it.
4 min
How to Monitor an AI Agent in Production
Your agent can be HTTP-200, fast, and cheap while being completely wrong. The metrics that keep a web app healthy are blind to the ways an agent actually fails.
5 min
GAIA2: The Agent Benchmark Where the Clock Never Stops
Static benchmarks freeze the world while an agent thinks. Meta's GAIA2 lets time run — and the smartest model, GPT-5, turns out to be the one that misses deadlines.
4 min
Cost-Aware Agent Evaluation: Why Your Benchmark Needs a Dollar Axis
An agent leaderboard that ranks only on accuracy is secretly ranking on willingness to spend. Add the cost axis and the board's #1 is often not even on the frontier.
5 min
τ-bench vs τ²-bench: The Agent Benchmark That Scores Whether You Can Guide a Human
Most agent benchmarks hand the whole task to the model. τ-bench keeps the user in the loop, and τ²-bench gives the user their own hands — which is where frontier agents quietly fall apart.
5 min
SWE-EVO vs SWE-bench: The Long-Horizon Test Coding Agents Fail
A new benchmark drops the same models from ~73% to ~25% — not by making the bugs harder, but by taking away the one thing SWE-bench always handed over: a map to the change.
4 min
SWE-bench Pro vs SWE-bench Verified: Why Top Coding Agents Dropped From 70% to 23%
The same models that ace SWE-bench Verified collapse on its successor. The gap isn't difficulty — it's the size of an illusion, and the only durable fix turned out to be a software license.
5 min
How to Evaluate a Deep Research Agent: BrowseComp vs DeepResearch Bench
The benchmarks for web-browsing agents split along a fault line the coding benchmarks never had — and the trick that makes one of them work quietly hides which half of your agent is actually good.
5 min
Terminal-Bench vs SWE-bench: Why Patching Code and Operating a Shell Are Different Skills
SWE-bench hands an agent a broken test and a healthy repo. Terminal-Bench hands it a live machine and lets it break things. That's why a top SWE-bench score tells you almost nothing about the second number.
5 min
Recovery-Bench: Why Top Agents Still Fail to Recover From Their Own Mistakes
A new benchmark replays an agent's failures into a corrupted environment and asks a fresh model to fix them. The leaderboard reorders — recovery is not the same skill as solving.
4 min
Record and Replay Testing for AI Agents: Deterministic Tests Without Live LLM Calls
You can freeze an agent run and play it back in CI — but there are two layers you can record at, and picking the wrong one means your tests stop catching the bug you actually care about.
5 min
How to Test an AI Agent With Simulated Users (and Why the Fake User Is the Hard Part)
You can't script a conversation, so you hand the user's seat to a second LLM. That move doesn't solve your measurement problem — it relocates it into a simulator you never validated, and the default one grades on easy mode.
5 min
How to Roll Out a New LLM in Production: Shadow vs Canary vs A/B Testing
The progressive-delivery playbook assumes a bad release trips an alarm. A worse model returns HTTP 200 on time with a fluent wrong answer — so the canary you copied from your web service is blind to the only failure that matters.
6 min
How to Evaluate a Voice Agent: Why Text-Agent Metrics Miss the Real Failures
Transcription accuracy is table stakes. The failure surface that actually loses calls is conversational timing — turn-taking, barge-in, and an end-to-end latency budget you have to measure component by component.
6 min
How to Benchmark LLM Inference: Why One Tokens-Per-Second Number Is Lying to You
A single throughput figure is uninterpretable without the load that produced it and the prompt shape you fed in. The honest output of an LLM benchmark is a curve, and the number that matters is goodput — the most traffic you can serve while still meeting your latency SLO.
5 min
How to Add LLM Evals to CI/CD Without Building a Flaky Gate
You wire your eval into GitHub Actions, gate the merge on it, and a week later it's red on a PR that changed nothing. The fix isn't a retry — it's admitting an eval is a measurement, not an assertion.
5 min
Pass@k vs Pass^k: Measuring Whether an Agent Is Reliable, Not Just Capable
pass@k asks whether an agent can ever solve a task. pass^k asks whether it solves it every single time. For long-horizon agents those are different questions — and the gap is where production failures live.
5 min
OSWorld vs WebArena vs WebVoyager: How to Read a Computer-Use Agent Benchmark
Three benchmarks, three verification methods, three very different definitions of 'success' — so a single computer-use percentage tells you almost nothing without the asterisks.
4 min
How to Debug an AI Agent
Print statements debug code. But the agent's code did exactly what it was told — the bug is in the context the model saw and the decision it made there. You debug an agent by reading transcripts, not by stepping through functions.
4 min
Braintrust vs Arize vs Opik: Choosing an LLM Eval Platform in 2026
The eval-tooling field just split into three camps and lost two players to acquisition in a single month. Pick on philosophy and independence, not the feature grid.
3 min
Agent-as-a-Judge vs LLM-as-a-Judge: Grading the Trajectory, Not Just the Answer
An LLM judge scores the final answer. For a multi-step agent, that signal is sparse, late, and easy to fool — a broken trajectory can still land on a right answer, and you'd never know.
5 min
Langfuse vs LangSmith vs Braintrust: LLM Observability and Evals Compared
Three platforms that look like competitors but optimize for different primary jobs, with lock-in profiles that diverge sharply once you read the fine print.
4 min
Online vs Offline Evals for AI Agents: Why Production Traces Need a Different Scorer
Offline evals ask whether the agent matched a known answer. Online evals can't — there is no answer. Treating them as one pipeline with one metric is the mistake that lets agents pass every test and still fail in production.
4 min
How to Reduce LLM Hallucinations in Production
You can't prompt a model into never being wrong — hallucination is the same machinery as a correct answer. The win is making every claim cheap to check.
6 min
How to Get a Confidence Score From an LLM (and Why the Easy One Lies)
Token logprobs are right there in the API, cheap and ignored — and after RLHF they're systematically overconfident. The signal that actually tracks whether the answer is right costs you N times the inference.
5 min
How to Build an LLM Eval Dataset
The scoring framework is the commodity. The hard, valuable, un-buyable work is looking at your own outputs and distilling real failures into labeled cases — your eval set is a precipitate of error analysis, not a download.
4 min
How to Detect LLM Hallucinations: Faithfulness Is Not Factuality
Almost every hallucination detector measures one thing — whether the answer is grounded in the context it was given. That is not the same as whether the answer is true.
4 min
garak vs PyRIT vs promptfoo: Which LLM Red-Teaming Tool to Actually Use
Three open-source tools dominate LLM red teaming — but they aren't rivals. One scans a model, one is a framework for building attacks, one is a CI gate. Pick by layer.
4 min
How to Evaluate an AI Agent's Tool Use, Not Just Its Answer
There is rarely one correct path through a task, so grading an agent against a golden trajectory fails. Grade invariants over the path, and the final state, instead.
5 min
Prompt Management: Langfuse vs PromptLayer vs Agenta (and Why a Registry Isn't Enough)
A prompt registry lets you change prompts without a deploy. On its own, that just lets you change them faster — not better. The tools that compound tie every version to an eval.
4 min
SWE-bench vs τ-bench vs GAIA: Which Agent Benchmark Actually Predicts Production
They look like a difficulty ladder. They're three orthogonal axes — and only one of them measures the thing that decides whether your agent survives contact with real users.
4 min
OpenLLMetry vs OpenInference: OpenTelemetry for LLM Agents in 2026
Both libraries emit OpenTelemetry spans for your agent. They disagree on what to name the attributes — and that disagreement, not the instrumentation, is your real lock-in.
5 min
DeepEval vs Ragas vs Promptfoo: Choosing an LLM Eval Framework
Three popular eval frameworks that look interchangeable answer three different questions — pick the one that matches the question you actually have.
5 min
LLM-as-a-Judge: How to Build an Eval That Doesn't Quietly Lie to You
Using a model to grade your model feels like measurement. Until you learn what the judge is actually rewarding — verbosity, position, and its own prose — it's closer to a focus group of one.
5 min
Langfuse vs LangSmith vs Arize Phoenix: Choosing LLM & Agent Observability in 2026
The real choice isn't which dashboard looks nicer — it's what unit of work you trace and who owns the trace data after the agent finishes.
5 minLatest in Evals & Observability
Not buyer's guides — the news, teardowns, and explainers behind this topic.
Tool Highlight: Langfuse — see what your LLM is actually doing, and grade it
What Langfuse is, who it's for, how to start in one docker command or one free signup, what it costs (as of July 2026), and the honest catch — the open-source observability-and-evals layer for founders shipping AI features.
4 min
The $510B Half: Exits Reopen, Compute Gets Its Neocloud, and Accenture Springs a Leak
H1 2026 closed at a record $510B in venture funding; this week Together AI raised $800M, Figma bought a vibe-coding team, Bending Spoons IPO'd up 40%, and a hacker claimed 35GB of Accenture source code. Read for founders.
5 min
Agents' Last Exam: Frontier Agents Pass 2.6% of Hard Professional Work — and the 2.6% Is the Point
Berkeley's ALE scores whole deliverables, all-or-nothing, the way a client would. That single methodology choice is why the number is 2.6% and not the 90s vendors keep quoting.
4 min
The Trace Is the New Log
Agent observability didn't invent a standard. It surrendered to a boring one from 2019 — and in doing so quietly retired the log as the unit of truth.
5 min
The Evals Are the Product
Agents got trivial to build and impossible to trust. The repos worth starring now aren't frameworks — they're the eval and tracing layer that tells you whether the thing actually works.
6 min
The Benchmarks Are Theater Now
When every frontier model clusters within a tenth of a point on the same saturated tests, the leaderboard stops measuring quality and starts measuring marketing.
4 min