Promptfoo and Phoenix land in the same search queries, the same "LLM eval tools" listicles, and the same Slack threads — so builders keep asking which one to pick, as if they were two brands of the same thing. They aren't. They sit at opposite ends of the eval timeline: one is a gate you stand in front of a deploy, the other is a microscope you point at production. Pick by where your quality problem lives, not by feature checklist — and if you have both problems, you run both. Here's the clean split.

The one-line version#

Everything below is downstream of that sentence.

Promptfoo: the gate#

Promptfoo is declarative and offline. You write a YAML file that names your prompts, your providers (60-plus, from the hosted APIs to a local Ollama), and a set of assertions that score each answer. Then:

npx promptfoo eval

It runs every prompt-and-model combination against every test case and scores them side by side. The detail that matters for a solo builder: promptfoo eval exits non-zero when assertions fail. That single fact is what turns it into a CI quality gate — drop it into a pipeline step with your API keys as secrets, and a regression in your prompt blocks the merge exactly like a failing unit test. Its other half is a red-team mode that generates adversarial prompts across vulnerability classes (prompt injection, jailbreaks, data leakage, excessive agency), with presets aligned to the OWASP LLM Top 10 — so the same gate that catches quality regressions also catches "did my last prompt change reopen an injection hole." This is the same lane as deepeval vs. ragas vs. promptfoo and garak vs. pyrit vs. promptfoo — the pre-ship, test-set-driven end of the world.

What Promptfoo can't do: tell you what happened to a real user yesterday. It only knows the cases you declared.

Phoenix: the microscope#

Phoenix, from Arize, starts from the opposite premise — that the interesting failures are the ones you didn't anticipate, and they're already sitting in your production traces. It ingests OpenTelemetry / OpenInference spans from your running agent, then runs phoenix.evals evaluators over them:

import phoenix as px
from phoenix.otel import register

register(auto_instrument=True)     # your agent's calls now emit spans

# ...run the agent, then pull spans and judge them...
from phoenix.evals import HallucinationEvaluator, QAEvaluator, run_evals
results = run_evals(dataframe=spans_df,
                    evaluators=[HallucinationEvaluator(judge_model),
                                QAEvaluator(judge_model)])

Because the evals are anchored to spans, the LLM-judge's score and its written explanation attach back onto the trace. So when a user reports a bad answer, you open that exact request, see every step the agent took — retrieval, tool calls, the final generation — and read why the judge scored it low, right there on the span. That is a debugging superpower a test-set tool structurally cannot have. Phoenix sits next to the observability stack we compared in Langfuse vs. LangSmith vs. Phoenix — but the point here is the evals on top of those traces, not the tracing alone.

What Phoenix won't do for you: block a bad deploy before it reaches anyone. It watches; it doesn't bar the door.

They're one loop, not a fork#

The reason "which one?" is the wrong question: the tools compose into a single quality loop.

  1. Gate the pull request with Promptfoo. No prompt or model change merges without clearing your assertions and surviving the red-team scan.
  2. Watch production with Phoenix. Real traffic hits real edge cases; Phoenix's trace-anchored evals surface the ones your test set never imagined.
  3. Feed the misses back. Every genuine failure Phoenix finds becomes a new Promptfoo test case, so the gate gets stricter exactly where reality proved it was too loose.

Skip step 1 and regressions ship. Skip step 2 and you're blind to everything you didn't predict. The mature setup runs both, and the arrow from Phoenix back into Promptfoo's test set is what makes the whole thing get better over time instead of just louder.

The 2026 ownership footnote#

One thing changed under Promptfoo this year that's worth knowing before you standardize on it: OpenAI announced it would acquire Promptfoo on March 9, 2026, folding it into OpenAI's enterprise-agent platform — and stated that Promptfoo remains open source under its current license (OpenAI; Promptfoo). For a team of one the practical read is unchanged: it's still a free CLI you run in your own pipeline. Phoenix remains Arize's open-source project. Neither fact should move your decision — that decision is still made by where your quality problem lives: at the gate, under the microscope, or, most likely, both.