If you're calling an LLM from production and you can't answer "how many requests did that feature make yesterday, and what did they cost?" — this is the fastest fix I know of, and it's free.
The one-line answer: Cloudflare AI Gateway is a free proxy that sits between your app and any LLM provider; you route your calls through it by changing your SDK's base URL to https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/{provider}, and in return you get caching, rate limiting, retries, fallbacks, logs, and cost analytics without touching your application logic.
What it is#
AI Gateway is a control plane for your model traffic. Instead of your app talking directly to api.openai.com or api.anthropic.com, it talks to a Cloudflare endpoint that forwards the request upstream and records everything that happens: the request, the response, token counts, latency, errors, and cost. Because it's a transparent proxy, it speaks each provider's native API — you don't adopt a new SDK or rewrite your prompts. You point one URL somewhere new and keep everything else.
That positioning matters. A lot of "LLM ops" tools ask you to route your code through their library and their abstractions. If you want a full multi-provider router with load balancing and key management rather than a transparent proxy, that's a different tool class — see our comparison of Bifrost vs LiteLLM vs Portkey. AI Gateway sits one layer earlier: it observes and protects whatever calls you already make. AI Gateway is closer to plumbing: it's a pass-through that happens to observe, cache, and protect the pipe. The features it layers on include response caching, rate limiting (fixed or sliding windows), automatic retries and model fallbacks, per-request and per-model cost tracking, spend limits that block requests once a dollar budget is hit, request/response logging, guardrails, and evaluations. There's also a unified, OpenAI-compatible endpoint that lets you hit many providers through a single URL and switch models with dynamic routing.
Who it's for#
This is squarely aimed at solo builders and small teams shipping AI features on a real budget. If you're a founder whose app makes a few thousand model calls a day, the two things that will hurt you first are a surprise bill and a provider outage — and AI Gateway addresses both. Caching cuts spend on repeated prompts, spend limits cap the worst-case blast radius, and fallbacks keep you serving when one provider hiccups.
It's also for anyone who is currently flying blind on observability. The moment you have more than one AI feature, the dashboard's per-model breakdown of volume, errors, and cost is the difference between guessing and knowing. (If you need deeper trace-level debugging of what the agent did rather than gateway-level metrics, that's a job for a dedicated tracer — see Langfuse vs Laminar vs Arize Phoenix; the two layers complement each other.) You don't need to be on Cloudflare for the rest of your stack to use it — the gateway works with a hosted app anywhere.
If you need enterprise log retention, SSO-gated audit trails, or you're pushing tens of millions of requests, you'll bump into the metered limits below and want to evaluate the paid tier or a dedicated platform. For most people reading this, the free tier is more than enough.
How to start#
Two steps. In the Cloudflare dashboard, create a gateway — you get an account ID and a gateway ID. Then change your SDK's base URL. That's the whole migration.
Before (talking directly to Anthropic):
from anthropic import Anthropic
client = Anthropic(api_key="sk-ant-...")
msg = client.messages.create(
model="claude-opus-4-8",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello"}],
)After (same code, routed through the gateway):
from anthropic import Anthropic
client = Anthropic(
api_key="sk-ant-...",
base_url="https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/anthropic",
)
msg = client.messages.create(
model="claude-opus-4-8",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello"}],
)The OpenAI version is identical in spirit — swap /anthropic for /openai:
from openai import OpenAI
client = OpenAI(
api_key="sk-...",
base_url="https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/openai",
)Prefer not to touch code at all? Many SDKs read an environment variable, so you can set it at deploy time:
export ANTHROPIC_BASE_URL="https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/anthropic"Once traffic flows through it, you can turn on features per gateway (caching, rate limits, fallbacks) in the dashboard, or override behavior per request with headers — no redeploy required. If you want provider-agnostic calls, use the unified endpoint at .../{account_id}/{gateway_id}/compat and name your model as {provider}/{model}; the gateway routes it and you swap models by changing a string.
What it costs#
The headline: the core is free, and the gateway adds no per-call fee. You still pay your model provider directly at their normal token prices — Cloudflare isn't reselling inference here, it's observing it. Analytics, caching, rate limiting, retries, fallbacks, and cost tracking all come at no gateway charge.
The one thing that's metered is stored logs. On the Workers Free plan you get 100,000 AI Gateway logs per month across all your gateways; once you hit that, logging pauses until the next cycle. Upgrading to Workers Paid (from $5/month) raises the stored-log limit substantially and unlocks Logpush for streaming logs elsewhere. Note that caching well actually helps you here twice: a cache hit saves provider tokens and is cheaper on the logging side than a full round trip.
I won't pretend there are no tradeoffs. You're adding a network hop and trusting Cloudflare with your prompt and completion data (which is logged by design, though configurable). But for a free, drop-in way to stop guessing about your AI spend and to add resilience you'd otherwise hand-roll, it's one of the highest-leverage afternoon upgrades a solo builder can make. Change one URL, watch the dashboard fill in, then turn on caching and a spend limit before your next launch.
