The one-line version: OpenTelemetry's GenAI instrumentation is quiet by default — OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT defaults to NO_CONTENT, so your prompts and completions are not recorded until you flip it on. The problem is that you almost always do flip it on — the message content is what makes an agent trace worth reading. And the instant you do, every user message, every tool input, every API key someone pasted into a system prompt starts streaming to a third-party tracing vendor. The answer isn't to go dark. It's to redact in three layers so the trace stays useful and the secrets never arrive.

This is the privacy companion to our how-tos on sending agent traces to Honeycomb over plain OpenTelemetry and cutting the trace bill with tail sampling — and the trace-side sibling of redacting PII before it reaches the model.

Layer 1 — decide what you capture at all#

The cheapest data to secure is the data you never record. The GenAI instrumentations gate message content behind one flag, and it ships off. Treat that flag as a decision, not a default you inherit from a base image:

# off — you get span structure, timings, token counts, model names, no bodies
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT
# on — you also get prompt + completion bodies. This is where PII starts flowing.
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_AND_EVENT

Keep it NO_CONTENT in the services that touch the most sensitive input, and turn it on only where you genuinely debug prompt behavior. You still get the shape of every trace — the loop, the tool calls, the latencies, the token spend — without the bodies. That alone answers most "why did the agent do that" questions.

Layer 2 — scrub at the source, before the span exists#

Layer 1 is all-or-nothing per flag. When you do capture content, strip the secrets you can recognize before you build the span, in your own code. A signed URL, a bearer token, a card number in a support message — match and mask them at the point you set the attribute:

SECRET = re.compile(r"(sk-[A-Za-z0-9]{20,}|Bearer\s+[A-Za-z0-9._-]+|\b\d{13,16}\b)")

def safe(text: str) -> str:
    return SECRET.sub("[REDACTED]", text)

span.set_attribute("gen_ai.prompt", safe(prompt_text))

This is the only layer that keeps a secret from ever leaving your process — which also means it protects you when the same value would have hit your own request logs, and when your collector is misconfigured. It's yours to maintain, and it will miss things. That's what Layer 3 is for.

Layer 3 — the collector backstop that fails closed#

Put a central redaction processor in the OpenTelemetry Collector, in front of the exporter, so one policy covers every service:

processors:
  redaction:
    allow_all_keys: false
    # only these attribute keys survive; everything else is deleted first
    allowed_keys:
      [gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens,
       gen_ai.usage.output_tokens, gen_ai.operation.name, gen_ai.tool.name,
       gen_ai.prompt, gen_ai.completion]
    # mask secret-looking KEYS entirely (e.g. authorization, x-api-key)
    blocked_key_patterns: ['(?i).*authorization.*', '(?i).*api[_-]?key.*', '(?i).*token.*']
    # mask VALUES matching these regexes inside the keys you kept
    blocked_values:
      ['\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b',   # card numbers
       '\b[\w.+-]+@[\w-]+\.[\w.-]+\b']               # emails

The order is what makes it safe. Attributes not on allowed_keys are removed before any value check runs — so an attribute nobody vetted is dropped, not shipped. Then values are masked: whole values under a secret-looking key via blocked_key_patterns, and substrings matching a blocked_values regex inside the keys you kept. ignored_keys are processed first and always pass — reach for them only on keys you've proven are safe. This is the pattern Honeycomb documents for handling sensitive data and OpenTelemetry's own security guidance points at.

Why all three, not one#

Each layer covers the previous one's blind spot. Layer 1 is free but coarse. Layer 2 is precise and keeps secrets in-process but relies on matchers you wrote. Layer 3 is central and auditable — one place to prove to an auditor what leaves the building — but only sees data that already left your app. Ship one redaction policy at the collector so nothing depends on every service getting it right, scrub the known-shape secrets in-process so they never travel at all, and capture content only where you'll actually read it. The trace stays as useful as ever. The secrets just stop riding along.