The default answer to "where does my agent's model run?" has been "someone else's datacenter" for so long that most founders never ask the question. But 2026 quietly made it a real choice. Microsoft put a 14B tool-calling model inside Windows, Apple ships on-device agents through App Intents, and open small models keep getting more capable. So the question is no longer can you run the agent on the machine — it's when you should.

Here's the honest math, and it's simpler than the vendor decks make it.

It's not the token price — it's the shape of the cost#

A cloud small-model API and an on-device model have opposite cost curves.

A cloud API — take Gemini 3.6 Flash, reported around $1.50 per million input and $7.50 per million output tokens — has a $0 floor and a linear, uncapped climb. You pay for exactly what you use, which is wonderful at low volume and merciless at high volume: the cost never stops scaling with traffic.

An on-device model is the mirror image: a fixed cost (hardware you already own, or amortize) and a $0 marginal cost per call. The thousandth call and the millionth call cost the same — nothing.

Two lines, one rising and one flat. They cross. Everything below is about where.

The cross-over is volume — on light, relentless work#

The calls that make on-device win aren't the impressive ones. They're the boring, high-frequency layer of any real agent: classify this file, pull these fields, decide which tool to call, route this request, summarize this page, redact this before upload.

A single such call — roughly 500 input and 150 output tokens — costs a small fraction of a cent on a cheap cloud tier. Trivial. But an agent doing this thousands of times an hour is making millions of these calls a month, and now that fraction of a cent is a real line item that grows with every new user. Locally, that same growth costs zero.

So the rule of thumb: the more light calls you make, the more the flat line wins. Somewhere past a few million routine calls a month, "free per call" beats "cheap per call" on cost alone — and it wins outright, at any volume, the moment the data can't leave the machine or the app has to work offline.

On-device isn't the cheap option for a small team. It's the zero-marginal-cost option — which only matters if you make a lot of calls, or if the data was never allowed to leave.

Where on-device loses — and should#

Don't romanticize it. A small local model is a workhorse and a router, not a genius. Send it hard multi-step reasoning and it will underperform a frontier tier. Ask it to hold a 200k-token context and the local KV cache becomes your bottleneck. Need the agent to run on a customer's phone or a browser you don't control, and "install a 14B model" is a non-starter. For all of that, the cloud earns its price.

And "on-device" is never free of cost — it just moves the cost. You trade a variable, uncapped token bill for a fixed hardware cost plus an ops surface: uptime, updates, model management. A bootstrapped team should count that ops time as real.

The answer most teams land on: hybrid#

The 2026 pattern that actually holds up isn't either/or. It's a local model in front as a cheap, private router — fielding the easy, high-volume calls at zero marginal cost — that escalates only the genuinely hard calls to a cloud tier. Do that and your token bill starts tracking difficulty instead of traffic, which is exactly the curve you want when every new user currently adds to the meter.

Do this: count your agent's calls for a week and sort them by difficulty. If the bottom 80% are light, repetitive, and — especially — touching data you'd rather not ship to a third party, that's the slice to move on-device. Leave the hard 20% in the cloud. The bill you save is the flat line doing its job.