Short version: By default, serverless inference bills you the real-time rate for every token — even the ones no human is waiting on. Both Together AI and Fireworks AI will cut those async tokens by 50% if you submit them as a batch job. Same model, same output, half the money. Most teams never flip the switch because their code funnels everything through the real-time endpoint. Here's how to split the traffic and claim the discount.
The one question that sorts your traffic#
Before any code, sort your LLM calls with a single test:
Does a human wait on this response?
- Yes → keep it real-time. Chat, agent tool loops, autocomplete, live search. Latency is part of the product.
- No → send it to batch and halve the bill.
That "no" bucket is bigger than most teams think. It usually includes:
- Nightly or scheduled summarization of documents, tickets, transcripts
- Bulk classification / tagging / extraction over a corpus
- Embedding backfills when you add a field or re-index
- Eval and benchmark runs (these are pure batch — nobody's watching)
- Synthetic-data generation for fine-tuning or tests
- Reprocessing an entire corpus after you change a prompt
If any of that is on your bill at real-time rates, you're paying exactly 2× for it.
What the discount actually is#
Neither provider makes you negotiate. It's automatic and it's the same model:
| Together AI | Fireworks AI | |
|---|---|---|
| Batch discount | 50% | 50% |
| Applies to | Async jobs | Both input and output tokens |
| Processing window | Up to 24 hours | Async return (not real-time) |
| Model / quality | Unchanged | Unchanged |
You are not buying a worse model or a smaller context. You are trading latency you don't need for money you'd rather keep. Together processes the job within a 24-hour window; Fireworks returns results asynchronously rather than in real time. For a nightly job that runs at 2 a.m. and is read at 9 a.m., that window is free.
The wiring: split at the queue, not in the model call#
The mistake is checking "is this batchable?" deep inside your model wrapper. Do it at the queue boundary, where you already know whether a task is interactive.
- Split the lane. When work is enqueued, tag it
interactiveordeferred. Interactive work goes straight to the real-time endpoint. Deferred work accumulates. - Submit a batch. Serialize the deferred requests as JSONL — one request per line, each with a unique custom ID — and upload to the provider's batch endpoint. You get back a job handle.
- Poll for completion. Check the job handle until the results are ready inside the processing window. This is a background loop, not a blocking wait — nothing in your request path stalls on it.
- Reconcile by ID. Download the results file and match each output back to its originating task using the custom ID you assigned. Write results where the deferred task expects them.
The whole thing is a four-step loop — upload → poll → download → reconcile — worth wrapping in one small helper so any deferred task in your app can hand off with a single call. The custom-ID discipline is the part people skip and regret: batch results come back unordered, and the ID is the only thing tying an answer to its question.
Where batch quietly breaks#
Two failure modes, both avoidable:
- You batched something interactive. If a user is blocked on the output, a 24-hour window is not a discount — it's an outage. Keep the lane split honest.
- **You need the answer to drive the next request.** Batch is for independent requests. A multi-step agent loop where step 2 depends on step 1's output is not batchable as one job — that's a real-time chain. (You can still batch the fan-out leaves of the loop.)
The payoff#
This is the single highest-leverage cost lever on serverless inference, because it changes nothing about your model, prompt, or output — it only reclassifies traffic you were overpaying for. Wire the queue split once, and every deferrable token from then on costs half. For anything running a recurring offline job, that pays back the engineering hour almost immediately.
If you haven't yet decided which provider to run this on, we compared the serving options — serverless tokens vs dedicated GPU-hours — in Together vs Fireworks vs Baseten: where to actually serve your open-weight model.
