Here is a number worth sitting with before you buy another observability tool: 6 out of 14.
That is how many distinct ways an AI agent can fail that a stock OpenTelemetry trace can actually detect, according to AgentTelemetry, a benchmark and toolkit published this year in the proceedings of the 3rd ACM International Conference on AI-Powered Software. The authors did something the vendor comparison charts never do. Instead of asking which tool has the nicest waterfall view, they asked a prior question: of all the ways an agent breaks, how many can a given trace model even see?
The answer for vanilla OpenTelemetry is a Fault Detection Rate of 0.429 — six of fourteen fault types. The other eight are not detected badly. They are not detected at all.
The result that should stop you#
The obvious reaction is: fine, OTel is generic, that is why the GenAI semantic conventions exist. Instrument the LLM call and the tool call with gen_ai.* attributes and the number goes up.
It does not. In the benchmark, OpenTelemetry with the full GenAI attribute set scores the same 0.429.
That is the finding to internalize, because it contradicts how most teams are spending their instrumentation effort. Richer attributes on a span do not help you detect a fault in a phase for which you never opened a span. The six faults OTel catches are the ones observable through generic signals any HTTP client already emits — an error status, a token count, a tool name, a latency spike. The eight it misses live in the parts of an agent that do not look like an API call at all.
The gap between 0.429 and 1.000 is not a data-volume problem. It is a shape problem.
What the trace is missing#
AgentTelemetry's taxonomy defines nine span kinds. Three of them — LLM_CALL, TOOL_CALL, RETRIEVAL — already map onto OpenTelemetry GenAI operations. One, AGENT, extends an operation OTel half-covers. The remaining five are new, and they are precisely the phases OTel has no vocabulary for:
- PLANNING — the agent decides on a sequence of steps. A bad plan is a fault the moment it is committed, long before any tool errors.
- REASONING — the step between reading the context and choosing the action. Every tool call can succeed and the conclusion still be wrong; this is the phase that judging the trajectory rather than the answer exists to grade.
- GUARD_RAIL — the safety check. A guardrail that silently passes an unsafe action is a fault whose signature is the absence of an intervention.
- DELEGATION — one agent handing a goal to a sub-agent. Goals leak at the seam; without a span on the handoff, you see two healthy agents and a dropped objective.
- MEMORY — the read and write of persistent state. A stale or mismatched memory read is invisible to a trace that only records the tool that fetched it.
Add span kinds for those five and the benchmark's detection rate climbs to a perfect 1.000 — an upper bound the authors present as proof of structural completeness, not as a claim that detection is easy. The point is narrower and sharper: the ceiling on what you can catch is set by the shapes in your trace model, and OTel's model is missing five of them.
The scale behind the number is not a toy. The harness runs 2,940 configurations — 14 fault types × 5 observability conditions × 7 agent frameworks × 6 models — and ships as a pip-installable library with adapters for those seven frameworks. This is a controlled measurement, not a vibe.
Why the invisible eight are the expensive ones#
There is a temptation to shrug: the faults OTel misses are the exotic ones, the common failures are tool errors, so 0.429 covers the cases that matter. The production data cuts the other way.
Sherlocks.ai analyzed 73 production agent incidents across customer environments between January and May 2026. Tool-call failures were the most common entry point — but, in their phrase, they almost never travel alone. Failures propagate: a tool error becomes a bad plan becomes a hallucinated recovery becomes a corrupted memory write. The incidents that were cheap to resolve were the ones with reasoning traces. The ones without averaged an MTTR of 4.2 hours — more than four times longer than incidents where full decision-trace logging was in place.
Read those two studies together and the story closes. The faults OTel cannot see are exactly the ones in the phases — planning, reasoning, memory — whose absence quadruples your time to recovery. You are blind in the rooms where the expensive fires start.
The uncomfortable correction#
Earlier arguments — including one on this very masthead — told you to instrument the part of the OTel GenAI spec that holds still: the span tree is stable even while the attributes churn, so build on the tree. That advice is still right, with one correction the benchmark forces: stable is not the same as complete. The tree holds still partly because it stops short of the five spans where agents actually break.
So the move is not to wait for a working group. It is to stop treating the missing spans as someone else's roadmap item and start emitting them yourself — a planning span, a reasoning span, a memory span, named plainly, wrapping the phases your framework already runs and currently throws away. The benchmark did the hard part. It told you which five shapes to add. The difference between 0.429 and 1.000 is not a bigger bill. It is a trace that has room for the way your agent actually fails.
