Spot vs On-Demand GPUs: When Interruptible Instances Actually Cut Your Bill (and When They Torch a Training Run)
Spot GPUs are the same H100s at 60–90% off — until the provider reclaims one mid-job. The discount isn't the number that matters. The notice window is.
5 min
Muse Spark 1.2 Is Meta's Third Model in Four Months — and This Time the Whole Gain Is Agentic
Meta shipped Muse Spark 1.2 on August 5 at the same $1.25/$4.25 price as 1.1, but the three points it added on the intelligence index landed almost entirely in agentic work: its real-world-task Elo jumped 260 points and Terminal-Bench climbed to 82.9%. For founders, the question isn't whether it's frontier — it's whether a same-price, better-at-agents backend earns a slot in your router.
4 min
SkyPilot vs dstack: Two Ways to Run a GPU Job on the Cheapest Cloud That Has One
Both let you launch training, inference, or an agent job across any GPU cloud without lock-in. They disagree on what you're actually managing — a job, or your whole compute plane.
4 min
Serverless GPU vs Dedicated Instances: When Per-Second Billing Beats a Reserved H100
The whole decision comes down to duty cycle — how many hours a day your GPU is actually busy — and how much cold-start latency you can stomach. Here's the break-even line.
4 min
RunPod vs Modal vs Baseten: What Serverless GPU Actually Costs in August 2026
Renting a bare H100 by the hour is the wrong model for bursty agent inference — you pay for idle. Serverless GPU scales to zero and bills by the second. Here's what the three big platforms charge, and the billing detail that decides your invoice.
4 min
How to Fan Out Agent Tool Calls Concurrently Without Tripping Your Rate Limit
Your agent emitted eight tool calls in one turn. Running all eight at once is how you turn a fast turn into a 429 storm. The fix is a bounded semaphore, backoff that honors Retry-After, and returning every result in one message — about 30 lines.
4 min
Anthropic Put a Deny Switch in Front of Claude: Inference Hooks, and the 30-Line Server That Turns Them On
Launched August 5, inference hooks route every enterprise prompt through your own HTTPS server for an allow-or-deny verdict before the model ever sees it. Here's the wire protocol, a working server, and the fail-open gotcha that quietly lets prompts through.
6 min
SGLang 26.07 Puts Blackwell Ultra in One Container Pull — What It Changes for Founders Renting GPUs
NVIDIA's August container ships B300, GB300, DGX Spark, and Jetson Thor support behind a single pinned tag. The news isn't a faster kernel — it's that your serving image now spans a $3,499 desk box and a $300k rack without a rebuild.
4 min
North Mini Code vs Devstral Small 2 vs Qwen3-Coder-30B: The Open-Weight Coding Model That Fits on One GPU
Three small open-weight coders you can self-host on a single card. They aren't really competing on SWE-bench — they're competing on how much work a GPU can do per hour and how cheap that GPU is.
5 min
Cloudflare Agents Week 2026: What a Solo Founder Should Adopt — AI Gateway, Sandboxes, and Email for Agents
Cloudflare shipped 20+ launches in a week — model routing, sandboxed code execution, versioned agent storage, native email, and a cost API. Here's what a solo founder should actually adopt, and what to skip.
4 min
Claude's inference_geo Flag: What US-Only Inference Actually Guarantees — and the 10% It Costs
Flipping inference_geo to "us" pins where the model runs and adds 10% to every token — but it does not, by itself, pin where your data is stored. Those are two different knobs, and founders keep flipping the wrong one.
5 min
How to Cut Your Claude Bill With a Three-Tier Model Router (Haiku → Sonnet → Opus)
Send every agent call to the cheapest model that can do the job, and escalate only when a validator says the answer isn't good enough.
8 min
Two Anthropic Changes Break Agents in Production This Week — a Retired Model ID and a Sampling Param That Now 400s
On August 5, calls to claude-opus-4-1 stop working — no grace period. And on Opus 4.7 and later, setting temperature, top_p, or top_k at all now returns a 400. Both are one-line fixes if you catch them before your users do.
5 min
Where to Actually Rent a GPU to Serve an Open Model in 2026: CoreWeave vs Lambda vs Nebius vs RunPod vs Together
Comparing hourly GPU prices first is the rookie mistake — half these clouds don't sell you the thing you think you're buying. Here's the product shape of each, and the utilization math that decides between renting by the hour and paying by the token.
4 min
Multi-Tenant Data Isolation for an AI SaaS: The Five Places Customer Data Leaks
A tenant_id column keeps your rows apart. It does nothing for your vector store, your prompt cache, your agent memory, or your trace logs — four leak surfaces classic SaaS never had. Here's how to close all five.
4 min
How to Serve an Open-Weights LLM with vLLM in 2026: The Commands, the VRAM Math, and the Cost-Per-Million
One command starts the server. The VRAM formula tells you which open models you can actually run on a founder budget — and the cost-per-million math tells you when self-hosting beats just paying the API.
5 min
How to Actually Configure vLLM's KV-Cache Offloading (0.26): The Flags, the Sizing Math, and How to Tell It's Helping
The overview posts told you 0.26 grew a memory hierarchy. This is the hands-on version — the real flags, a KV-bytes-per-token sizing rule, and the three metrics that prove offload is helping instead of hurting.
7 min
Rent a GPU or Call an API? The Break-Even Math for Serving an Open Model in 2026
A rented H100 costs the same whether it runs flat-out or sits idle. A per-token API costs nothing when no one's calling it. That single difference — fixed vs variable — is the whole decision, and it has a number.
4 min
Together vs Fireworks vs Baseten: Where to Actually Serve Your Open-Weight Model
Kimi K3's weights are public, so the real question moved from 'can I run it' to 'who runs it for me.' Together and Fireworks sell you tokens; Baseten sells you GPU-hours — and that one difference, not the price-per-token, decides which is cheaper for your traffic.
4 min
How to Run a Local Agent Backend on LM Studio's OpenAI-Compatible Server
Point the OpenAI SDK at localhost, load a tool-capable model, and your agent loop runs on your own hardware with zero code changes. Here's the whole path — plus the three gotchas that decide whether tool calls actually work.
4 min
What It Actually Costs to Rent an H100, H200, or B200 in August 2026
The gap between the cheapest specialty cloud and a hyperscaler is now roughly 5–7× for the same GPU. Here is the published on-demand price map — and the three numbers that decide which column you belong in.
4 min
Batch Inference and the 50% Discount Most Teams Never Turn On
If any part of your LLM workload can wait a few hours, you're probably overpaying for it by exactly 2×. Together and Fireworks both cut async batch jobs by 50% — same model, same tokens, half the bill. Here's what qualifies, how to wire it, and the one latency rule that decides whether it fits.
4 min
vLLM Retired guided_json: How to Write Structured Outputs the New Way
If you self-host on vLLM, the guided_json / guided_choice request fields you copied from a 2025 tutorial are deprecated. The whole family now lives under one structured_outputs object — here's the copy-paste migration for the server and the offline API.
4 min
North Mini Code vs Qwen3-Coder-Next vs GLM-5.2: The Smallest Open Coder That Still Clears the Bar
Cohere's North Mini Code is a 30B/3B model that fits on one H100 in FP8 with no quantization gymnastics. It gives up a couple of SWE-bench points to Qwen and GLM — and buys back the simplest self-host on the board.
4 min
fal vs Replicate vs Modal: Which Serverless GPU Should Serve Your Generative-Media Model?
Three platforms every founder shipping image, video, or voice AI ends up comparing — and the real axis isn't price per hour. It's how much of the stack each one hands you, which quietly decides your bill, your cold starts, and how much code you own.
6 min
When Speculative Decoding Hurts Throughput: The Batch-Size Crossover, and How to Find Your Own
You turned on speculative decoding and your endpoint got slower. That's not a bug — it's the design. Spec decode trades spare compute for lower latency, and above a certain batch size you've run out of spare compute. Here's where the line is and how to measure yours.
5 min
What an AI Agent Actually Costs Per Task: A Unit-Economics Worksheet for Founders
The per-million number on a model's pricing page is the worst predictor of your bill. Three variables — cache hit rate, output-to-input ratio, and how many turns the loop runs — decide what an agent task actually costs. Here's the worksheet that turns them into a number.
4 min
Tool Highlight: Tinfoil — Confidential LLM Inference Your Cloud Provider Can't Read
The reason your enterprise deal stalls at 'we can't send customer data to an LLM' isn't the model — it's that you can only promise the host never sees the prompt. Tinfoil runs the model inside a hardware enclave with remote attestation, so you can prove it instead.
5 min
How to Run DSpark Speculative Decoding in SGLang 0.5.16 (the Draft Length Sizes Itself Now)
SGLang 0.5.16 shipped DSpark: a speculative-decoding scheme that stops guessing a fixed draft length and lets each verify window size itself from the draft's own confidence. Here are the three flags that turn it on and when it actually pays.
5 min
OpenAI Cut Terra and Luna on July 30. On the Sticker, Luna Is Now the Cheapest Agent Backend Alive — On the Bill, the Ranking Barely Moved.
The July 30 price cut took Luna 80% off and Terra 20% off, undercutting Gemini 3.6 Flash on paper by 6×. Here's the per-completed-task routing map that survives the discount.
4 min
Give Every Agent Tool Call a Deadline — and Cancel It Cleanly When It Blows It
An agent that awaits a tool call with no timeout will hang forever the first time a downstream API stalls. Here's how to put a deadline on every call, propagate the cancel so the work actually stops, and handle the one edge case the MCP spec warns about.
5 min
vLLM vs llama.cpp for Serving gpt-oss on Your Own GPU
Same open-weight model, two very different servers. One is a datacenter throughput engine; the other runs anywhere. Here's which one your agent backend actually wants — and the GGUF caveat to know first.
3 min
Tool Highlight: Vercel AI Gateway — One Key, Automatic Failover, Zero Token Markup
A single endpoint to hundreds of models, automatic retries when a provider errors, and spend visibility tied to your projects — at 0% markup on tokens. Here's what it is, who it's for, and how to send your first request.
3 min
Reroute Instead of Erroring When an LLM Key Hits Its Budget: LiteLLM Budget Fallbacks
When a customer burns through their model budget, don't 429 them — silently drop them to a cheaper model that still has headroom. Here's the per-key config in about 15 lines.
4 min
How to Run gpt-oss-120b on a Single 80GB GPU for an Agent Backend
OpenAI's open-weight workhorse fits on one H100 because of MXFP4. Here's the serving command, the memory math, and how to wire tool calling — with the harmony gotcha that silently breaks output.
4 min
How to Pick a gpt-oss-120b Inference Provider: Cerebras, Groq, SambaNova, or a GPU Cloud
The same open model runs ~3× faster on wafer-scale silicon than on a fast GPU cloud, and the switch is one base-URL change. So the real decision isn't the model — it's matching a provider's speed-vs-price curve to whether a human is waiting.
5 min
Why Your Agent's Real Cost Is Its KV-Cache Hit Rate — and Four Rules to Protect It
You've been watching token counts. The number that actually moves your bill and your latency is the share of your prompt that hits cache — and most agent designs quietly wreck it.
4 min
Kimi K3's Weights Are Already 4-Bit: Don't Re-Quantize Them, and Don't Serve Them on Hopper
The open weights that landed July 27 aren't a full-precision checkpoint you shrink to fit — they're the model as trained. MXFP4 quantization-aware training changes two self-hosting reflexes, and getting them wrong costs you quality or memory.
3 min
Kimi K3, GLM-5.2, or DeepSeek V4? The Open Coding Tier Reshuffled July 27 — Pick by License and Serving Cost, Not the Leaderboard
Kimi K3's weights landed and it took the open-weight crown on two benchmarks at once. For most founders that changes nothing: the decision is still license and serving cost, and on those K3 is often the wrong default.
5 min
How to Build a Deterministic Agent Router — With an LLM Escape Hatch for the One Fork That Needs It
Most multi-agent routing is a lookup you already know at author time, billed back to you as a model call. Here's how to route with plain conditions, and spend a token only on the one branch that's genuinely ambiguous.
7 min
Anthropic Just Drew the Open-Weights Line — And Founders Running Kimi K3 Are on the Safe Side of It
Dario Amodei's July 27 essay calls non-dangerous open models 'a public good' and aims its three real asks at chips, distillation, and frontier safety-testing — none of which touch a team self-hosting an open model in production.
3 min
How a 2.8-Trillion-Parameter Model Stays Cheap to Serve: Kimi K3's Delta Attention and Attention Residuals
Kimi K3 is huge on paper and light on the meter — sparse MoE fires ~16 of 896 experts per token, Delta Attention bounds the long-context decode, and Attention Residuals is a training-time freebie.
6 min
You Can Now Download Kimi K3. Here's What It Takes to Serve 2.8T Open Weights Yourself
The weights dropped today. The headline is 2.8 trillion parameters; the number that sets your bill is 50 billion. Here is the real hardware math, the serving shape, and the one line that decides whether you rent or own.
4 min
vLLM 0.26 vs SGLang 0.5.16: The Sync Stall Is Settled — Now It's Spec-Decode and Prefix Caching
Both inference engines shipped the same day again (July 25). The scheduler-overlap fight that defined the last round didn't get a sequel — so the real question moved to speculative decoding, prefix caching, and which new models you can serve day one.
4 min
vLLM 0.26 Shipped: The Three Serving Knobs Worth Turning, and One Model List Worth Reading
The July 25 release adds fp32 lm_head via head_dtype, a different attention backend per KV-cache group, and an object-store tier for KV offload. If you self-host inference, here's what to flip and what it buys.
4 min
One GPU, Apache 2.0, No Vendor: Self-Hosting Qwen3.6-35B-A3B in July 2026
A 35B model that thinks like a small one: because only ~3B parameters fire per token, a quantized Qwen3.6-35B-A3B fits on a single 24GB card. Here's the exact serving command, the VRAM math, and the point where the API is still cheaper.
4 min
Fireworks Raised $1.5B at $17.5B — and 95% of Its Tokens Prove the Frontier Model Isn't What Production Wants
The inference platform's Series D isn't the story. The story is the number buried in it: 95% of the 40 trillion tokens it serves daily come from small, customized models — not the frontier flagships. That's the founder signal.
4 min
vLLM vs SGLang in 2026: The Real Decision Isn't Throughput — It's Your Prefix
Both engines killed the sync stall the same week, so peak tokens/sec has converged. The choice that actually moves your bill now is workload shape: does your traffic replay a big shared prefix every turn, or do you just need whatever model dropped this morning to run on the GPU you have?
4 min
OpenRouter vs LiteLLM vs Cloudflare AI Gateway: Marketplace, Proxy, or Edge — How to Route Your LLM Traffic in 2026
One buys you a marketplace, one is a proxy you run, one wraps the providers you already use. Here's how a founder picks where to put the LLM control plane in 2026.
5 min
How to Give Every User Their Own LLM Budget: Per-Key Spend Caps with LiteLLM Virtual Keys
Run one self-hosted LiteLLM proxy that mints a capped API key per customer, enforces rate limits, and tracks cost per key over a Postgres database.
4 min
Inference Chips Just Became Loan Collateral: What General Compute's $400M Deal Means for Your Token Bill
For the first time, a lender underwrote AI infrastructure against inference silicon instead of Nvidia GPUs. That's a signal about where cheap capacity is heading — and it points at your serving costs.
3 min
Etched Raised $300M for a Chip That Only Runs Transformers — and That's the Whole Bet
The Sohu ASIC claims 20× an H100 on inference by deleting everything that isn't a transformer. For founders, the number that matters isn't the speedup — it's what fixed-function silicon does to your token bill.
4 min
OpenAI Presence: The Agent-Ops Layer Is Now the Product — and It's White-Glove, Not Self-Serve
On July 22 OpenAI shipped Presence, a managed platform for running production agents — policies, guardrails, simulations, evals. The tell isn't the features. It's that you can't buy it with a credit card.
4 min
NVIDIA's Cosmos 3 Edge Puts a 4B Open World Model on One GPU — and Software Founders Should Read the Trend Line
NVIDIA shipped a 4-billion-parameter open world model that runs real-time robot control on a single GPU, no cloud. You probably aren't building robots — but the pattern (small, specialized, open, on-device) is the same one reshaping your model bill.
3 min
Fireworks AI Is Now a $17.5B Inference Cloud — What the Layer You Rent Just Told You
Nvidia-backed, reportedly north of $1B in annualized revenue and ~40 trillion tokens a day. The valuation isn't the story for a founder — the consolidation of the layer you serve open models on is.
3 min
Cloudflare AI Gateway: The Free Proxy That Caches, Rate-Limits, and Observes Every LLM Call
Point your existing OpenAI or Anthropic SDK at one new base URL and get caching, rate limits, retries, and cost analytics for free.
5 min
How to Put Spend Caps and Rate Limits on an AI Agent: The Three Layers That Stop a Runaway Bill
A looping agent can spend a month's budget in an afternoon. The fix isn't one setting — it's three independent brakes: a provider cap, a gateway budget, and a hard limit on the loop itself.
5 min
LM Studio Bionic: A Private, Local Agent for Open Models — What It Is, Who It's For, How to Start
LM Studio shipped a standalone agent app on July 16 that runs open models on your own machine: repo-aware coding, document work, and local voice input, with a zero-data-retention cloud option for the heavy jobs. If sending code or client files to a hosted API is a blocker, this is the founder's local-first path.
4 min
Both Your Inference Engines Shipped This Week. vLLM 0.25 and SGLang 0.5.15 Won the Same Fight.
vLLM deleted the CPU–GPU sync in the model runner; SGLang deleted it in the speculative-decoding scheduler. The frontier of serving throughput in mid-2026 isn't a faster kernel — it's the war on the stall.
4 min
How to Cache Your Agent's Tool Definitions and Cut Token Cost
Your tool schemas are the fattest, most stable block in every agent request — and the single highest-leverage thing to cache. The trick is not breaking the prefix.
4 min
The Money Moved to Agent Reliability: Three July Rounds That Show Where 2026 Capital Is Going
In a two-week stretch, the biggest agent checks skipped foundation models and landed on the reliability layer — evaluation, oversight, and domain decisioning.
4 min
Route Around a Flaky Model: A Fallback + Cost-Cap + A/B Router in ~60 Lines
You want to trial a cheap new model in your agent without a bad night. Here's a provider-agnostic router — primary plus ordered fallbacks, a hard cost cap, and a canary that logs cost-per-task — that drops in front of any OpenAI-compatible endpoint.
8 min
Every Model Tier Got Cheaper in Two Weeks: A Founder's Routing Map for July 2026
Between June 30 and July 9, Anthropic, OpenAI, xAI, Meta, and Google all shipped or repriced a model aimed squarely at cost-sensitive builders. Here's the whole board on one screen — and which lane to route each job to.
3 min
How to Build a Cost-Aware Model Router for Your Agent
Most agent turns are easy. Sending every one to a frontier model is the biggest bill you can cut without touching quality — here is the code.
5 min
How to Force Valid JSON From a Local LLM: Constrained Decoding in vLLM and SGLang
Prompt-and-pray parsing breaks in production the day a model emits a trailing comma. Constrained decoding makes invalid output structurally impossible — and it's usually faster, not slower. Here's the working setup, end to end.
5 min
How to Build a Fallback Model Chain: Route to a Cheap Model, Backstop with a Frontier One
Send most of your traffic to a cheap model and only pay frontier prices when something actually breaks. Here's the retry, timeout, and validation-gate code that makes that safe.
6 min
GLM-5.2 vs MiniMax M3 vs Kimi K2.7: Which Open-Weight Coder to Route To
Three Chinese labs, three different bets on the agentic-coding frontier — and the routing decision for a small team hinges on context length, multimodality, and license terms, not the leaderboard number.
5 min
How to Add a Fallback Model Chain Without Silently Degrading Quality
A fallback chain turns a 503 into a 200 — which is exactly the problem. The request succeeds on a weaker model, the answer gets worse, and nothing in your logs says so.
4 min
How to Add a Cheap-Model Fallback to Your Agent: Route Easy Work Cheap, Escalate the Hard 20%
One OpenAI-compatible client, two base URLs, and a fallback wrapper: send the bulk of your agent's calls to a cheap open-weight model and escalate only the calls that fail. A copy-paste pattern in ~40 lines of Python.
3 min
Where to Actually Serve an Open Model: Together vs Fireworks vs Baseten vs Modal vs DeepInfra
The five providers a founder actually chooses between all serve the same open weights. The decision isn't the provider — it's one axis: pay per token, or rent the GPU by the hour.
4 min
vLLM Sleep Mode: How to Free GPU Memory Between Agent Turns Without Reloading the Model
An idle agent still holds the whole GPU. Sleep mode parks the weights in CPU RAM and hands the VRAM back in under a second — so one card can run the model you're not using right now.
5 min
Tool Highlight: LiteLLM — One OpenAI-Shaped Door to 100+ Models
What LiteLLM is, who it's for, how to start (SDK in one line, self-hosted gateway in two), what it costs, and the honest catch — the open-source LLM gateway that lets you swap providers with a string change instead of a rewrite.
4 min
Terra vs Muse Spark 1.1 vs Grok 4.5: Which Cheap Agent Model to Route To
Three sub-frontier models launched inside 48 hours, all aimed at agentic and coding work, all undercutting the flagships. The one with the lowest sticker price is not automatically the cheapest to run — here's the decision, by the number that actually bills you.
5 min
Rate Limiting Your Own API: Token Bucket vs Sliding Window vs Fixed Window
Four algorithms, one question — do you want to smooth traffic, count it fairly, or forgive a burst? Pick the one whose flaw you can live with.
8 min
How to Cancel an LLM Request When the Client Disconnects — and Stop Paying for Tokens Nobody Reads
A user closes the tab mid-stream. Your server keeps generating to the last token, billing GPU time to output that reaches no one. Here's why abandoned streams keep running, and how to make the disconnect actually abort the request.
5 min
Bifrost vs LiteLLM vs Portkey: Picking an LLM Gateway After the 2026 Shakeout
TensorZero shut down, Helicone froze, Portkey got acquired, and LiteLLM shipped malware to PyPI. The gateway you pick in 2026 is a runtime and supply-chain decision — here's the one that changed the math, with the config to swap in.
4 min
This Week the Money Went to the Open-Model Stack: Ollama, Nemotron 3, and the Bet on Agent Reliability
Three moves in five days — a $65M raise, a family of open models with a 10x-cheaper agent story, and $40M for training environments — all point at the same shift: open weights are commodity, the edge is everything around them.
4 min
Tool Highlight: Ollama — the OpenAI-Compatible Seam Between Your Laptop and the Cloud
It started as 'run Llama on your Mac.' In 2026 it's how a small team runs open-weight models — Kimi, GLM, DeepSeek, Qwen — locally or hosted, behind the same API your code already speaks. Fresh off a $65M round.
3 min
The Run-Anywhere Week: ZML's Free Cross-Chip Server, OpenCode at 7.5M, and the Portability Bet for Founders
In one week the counter-move to walled, expensive inference got concrete — a free server that runs open models on any major chip, and a model-agnostic coding agent crossing 7.5M developers. Here's the founder's read on portability as a hedge.
5 min
How to Build a Cheap, Resilient Image-Generation Pipeline (Cache + Provider Fallback) in 2026
Now that AI images cost cents per thousand, the constraint isn't the model — it's the plumbing. Here's a copy-paste pipeline that caches by prompt hash, falls back across providers, and caps your spend before the invoice does.
6 min
How to Measure What an LLM Actually Costs You: Tokens, TTFT, and Throughput in Code
A rate card can't tell you cost-per-task — token counts and latency can, and this week's launches proved why. Forty lines of Python to measure the numbers that decide your bill.
4 min
Tuning Chunked Prefill in vLLM: The One Knob That Trades First-Token Latency for Smooth Streaming
max_num_batched_tokens looks like a throughput setting. It's really a fairness dial between the one user who pasted a novel and everyone else's token cadence.
3 min
TensorRT-LLM Is Removing the TensorRT Backend — the PyTorch Runtime Won
The library named after TensorRT is deleting TensorRT. The June 30 release candidate is the last to support the compiled engine backend; the next version removes it. The lesson isn't about NVIDIA — it's about which tradeoff keeps winning.
4 min
Serving DeepSeek V4: Why the Day-0 Recipe Matters More Than the MIT License
DeepSeek shipped a 1.6-trillion-parameter model under MIT and let vLLM and SGLang publish the serving recipes the same day. The weights are free and portable. The throughput that makes them economical is neither.
4 min
How to Set the Prefill-to-Decode GPU Ratio for Disaggregated Inference
Once prefill and decode live on separate GPU pools, you have to decide how many of each. The number isn't a property of your model — it's a property of your traffic, and it drifts.
4 min
Why Prefix Caching Quietly Fails in Agent Loops — and What Non-Prefix KV Reuse Does Instead
The universal advice is 'front-load your static system prompt so it gets prefix-cached.' In a tool-using or RAG agent, one mid-context insertion throws that whole cache away. CacheBlend keeps it anyway.
5 min
NIXL vs Mooncake: Choosing a KV-Cache Transfer Backend for Disaggregated Inference
Once you split prefill and decode onto separate GPUs, something has to ferry gigabytes of KV cache between them. NIXL and Mooncake are the two names you'll meet — and they aren't actually competitors.
4 min
How to Throttle an Agent Against a Third-Party API Rate Limit
The instinct is to rate-limit per user. An agent breaks that in one move: a single user's run fans out into hundreds of calls, and the ceiling that binds isn't yours — it's the API you're calling.
5 min
Together AI Raised $800M at an $8.3B Valuation. The Story Is What Agents Did to Inference Pricing.
A neocloud that owns none of the models it serves just booked $1.15B a year. The number that matters isn't the valuation — it's that open-model inference outgrew the labs whose weights it runs.
5 min
Retry Budgets for LLM Calls: Why Retrying Every 429 Makes the Outage Worse
Wrapping every model call in retry(3) feels responsible. Under a provider brownout it's the fastest way to turn a slowdown into a blackout. The fix is a budget, not more backoff.
4 min
How to Cancel a Running AI Agent — and Why Closing the Connection Doesn't Stop It
You press stop. Your socket closes. The GPU keeps decoding, the bill keeps climbing, and a half-finished tool call is still out there. Cancellation isn't a button — it's cooperation.
4 min
DeepSeek's DeepSpec Open-Sources the Hard Part of Speculative Decoding: Training the Draft Model
The speedup was never the bottleneck — the well-matched draft model was. DeepSpec ships the whole draft-training pipeline, MIT-licensed, with Qwen3 and Gemma as the default targets.
4 min
Can You Run an AI Agent on the Batch API? Mostly Not — and What to Batch Instead
An agent is a chain of steps that each depend on the last, so a 24-hour batch window can't sit on the critical path. You can't batch the loop — but the token-heavy work around it is exactly what batch was built for.
4 min
Anthropic Wants Claude on Microsoft's Maia 200 — Which Would Make It the Fourth Chip Running Inference
A reported deal to rent Azure servers full of Microsoft's inference silicon isn't about capacity. It's a tell about which half of an AI lab's compute is actually up for grabs.
5 min
The vLLM Semantic Router: The Gateway That Decides Whether to Reason at All
Every LLM router you know picks a model. This one runs inside Envoy's data plane and turns reasoning off — and on MMLU-Pro that raised accuracy 10 points while halving tokens.
4 min
RadixArk: Why NVIDIA, AMD, and MediaTek All Wrote Checks for the Same Open-Source Inference Engine
SGLang's team spun out as RadixArk on a $100M seed at a $400M valuation. Read the cap table, not the press release: hardware rivals rarely fund the same software unless it threatens something they all share.
4 min
How Many GPUs Does Your AI Agent Need? NVIDIA's NeMo Agent Toolkit Sizes the Cluster by Load Test, Not Math
The sizing calculator in NVIDIA's NeMo Agent Toolkit profiles a multi-agent workflow under concurrency and extrapolates a GPU count. The quiet lesson: an agent's cost is emergent, not calculable.
5 min
Why Your LLM Isn't Reproducible at Temperature 0 — and How to Fix It
Setting temperature to 0 doesn't make an LLM deterministic. The real culprit isn't sampling or 'random' GPU math — it's that your request's output depends on who else is in the batch.
4 min
Run a 671B Model on One 24GB GPU: The MoE Offload Trick, KTransformers vs llama.cpp
A frontier mixture-of-experts model has 671B weights but touches only ~37B per token. That gap is why you can serve DeepSeek-scale models on a single consumer GPU — if you split by tensor role, not by layer.
4 min
Multi-Region LLM Failover: Uptime Is the Easy Part — Your Cache and Your Data Residency Are the Bill
The managed cross-region inference you turned on already solved availability. It did it by dissolving the region boundary you may be legally required to keep — and by throwing away your prompt cache at the exact moment you're already degraded.
5 min
CoreWeave vs Lambda vs Nebius: How to Actually Pick a GPU Cloud in 2026
The neocloud pitch started as 'cheap raw GPUs vs AWS.' In 2026 the scarce input isn't price — it's powered, networked racks — and the category has quietly split into two businesses that barely compete.
4 min
vLLM Is Now a Startup: What Inferact Means for the Inference You Run On
The people who build vLLM raised $150M and became a company. The money isn't the story — who now sets the roadmap of an engine half the industry serves on is.
5 min
TPU vs GPU for LLM Inference in 2026: It Comes Down to the Network, Not the Chip
Per chip, Google's Ironwood and Nvidia's B200 are now within ten percent of each other on every number that used to decide this. The real fork is the interconnect — and vLLM just deleted the reason you couldn't cross it.
6 min
Text Generation Inference Is Archived: Migrating Off TGI in 2026
Hugging Face's TGI went read-only in March. The way it wound down — not the fact that it did — tells you where model serving actually settled.
4 min
EPLB vs LPLB: Why SGLang's 5x MoE Speedup Was a Solver, Not a GPU
SGLang v0.5.14 reports 5x throughput serving DeepSeek-V4 on GB300. The lever isn't Blackwell Ultra — it's a per-batch linear program that reroutes tokens across expert replicas. Static replication plans for the average; no batch looks like the average.
4 min
Prefix-Aware Load Balancing for LLM Inference: Why Round-Robin Wastes Your KV Cache
The load balancer you already trust is the wrong tool for a fleet of inference servers. Spreading requests evenly is exactly what destroys the cache that sets your latency and your bill.
6 min
OpenAI's Jalapeño Chip: The Real Bet Behind a Custom Inference ASIC
OpenAI's first silicon claims roughly 50% cheaper inference than Nvidia. The number is self-reported and unverifiable — but the vertical-integration bet underneath it is the part actually worth understanding.
4 min
How to Track AI Agent Costs in Production: Stop Counting Tokens, Start Counting Tasks
The per-token dashboard is lying to you. An agent's cost lives in the trajectory, not the request — and the only number that aligns finance with engineering is dollars per resolved task.
5 min
Why Your AI Agent Bill Grows Faster Than Its Workload: The Quadratic Nobody Prices In
Token prices are falling and agent bills are still exploding. The reason isn't the model getting pricier — it's that an agent re-pays for its entire history at every step, so cost scales with the square of the loop, not its length.
5 min
TensorZero Shut Down With Money in the Bank: What the LLMOps Squeeze Looks Like
An 11.7k-star, Rust-based open-source LLMOps stack archived itself on June 12 — not because it ran out of adoption or cash, but because the wedge it was built on is closing from both ends.
4 min
The Self-Hosted AI Gateway: 7 Open-Source Proxies That Became the Agent Control Plane
The 'AI gateway' stopped being a cost-tracking load balancer and turned into the policy layer for autonomous agents — and that shift is why the newcomers are all written in Go and Rust, benchmarking themselves against LiteLLM.
4 min
SGLang Makes Spec V2 the Default: Speculative Decoding Grows Up in v0.5.13
The headline in SGLang's June release isn't a speed number — it's a deprecation. Speculative decoding stopped being an expert knob and became the default path, and the old one is on the way out.
4 min
Claude Sonnet 5's Tokenizer Tax: Why the Same Rate Card Costs More Per Task
Sonnet 5's rate card matches Sonnet 4.6's — $3/$15 per million tokens. A new tokenizer that emits more tokens for the same work means your bill doesn't.
3 min
vLLM vs SGLang vs LMDeploy: Picking a Self-Hosted Inference Engine in 2026
With TGI archived and Hugging Face pointing everyone at vLLM and SGLang, the open-source serving field narrowed to three real choices. They hit nearly the same throughput ceiling from opposite directions — so speed is not the thing you're actually picking.
4 min
LLM Cascade vs Router: Escalate to a Bigger Model, or Route Around It?
A router picks a model before it sees the answer; a cascade tries the cheap one first and escalates only if a judge says so — and that judge, not the models, decides whether you actually save.
6 min
How to Enforce a Token Budget on an AI Agent (Not Just Measure It)
Most 'agent budgets' are alerts wearing a brake's uniform: they tell you after the money is gone. Real enforcement is a prediction problem, because the cost of the next step is a bound you can only ever estimate — never a number you can look up.
5 min
Batch API vs Real-Time Inference: The 50% Discount Isn't Why You Should Use It
Every provider now sells the same deal — hand over your requests, wait up to 24 hours, pay half. The savings are real, but the reason to reach for batch is the thing nobody puts on the pricing page.
4 min
How to Attribute LLM Costs Per Agent, Tenant, and Feature
The invoice arrives and no one can say which customer spent the money. Cost attribution isn't a report you run later — it's a schema decision you make at request time, and for agents the gateway total lies about where the spend went.
4 min
How to Handle a Truncated LLM Response: finish_reason, max_tokens, and the Reasoning-Token Trap
A cut-off completion isn't an error your code catches — it's a 200 OK whose only tell is a stop-reason field most callers never read. And on reasoning models, the fix everyone reaches for can hand you an empty response.
5 min
How to Deploy a Long-Running AI Agent Without Losing In-Flight Work
A web server drains its in-flight requests in 30 seconds and restarts. An agent's in-flight request is a multi-hour, side-effecting loop — so graceful shutdown stops being a deploy setting and becomes an architecture decision you had to make weeks earlier.
5 min
Kubernetes' Gateway API Inference Extension: When the Load Balancer Starts Reading GPU Metrics
Round-robin is the wrong way to route an LLM request. Kubernetes now has a GA'd standard that lets the gateway pick a model server by live KV-cache pressure and queue depth instead — and it changes what a load balancer is.
4 min
Cross-Cluster LLM Serving: Why KServe, llm-d, and Dynamo Stop at the Cluster Line
The Kubernetes-native serving stack got very good at spreading a model across a cluster. But in 2026 your GPUs aren't in one cluster — they're scattered across clouds by price and availability, and that's a different problem.
4 min
Spot GPUs for LLM Inference: How to Cut Serving Cost Without Dropping Requests
Interruptible GPUs scare people because of training horror stories. For stateless inference the math inverts — there's nothing to checkpoint, so the only real tax is cold start.
5 min
Provider-Agnostic AI Agents: The Lock-In Isn't Where You Think
Swapping LLM providers in one line is true for a chatbot and a lie for an agent. The cage is one layer up, in tool-calling behavior — and no gateway unlocks it for you.
5 min
Autoscaling LLM Inference on Kubernetes: Scale on the Queue, Not the GPU
The metric you'd reach for first — CPU, then GPU utilization — is the one that lies. A 70B pod can read 5% CPU and a calm GPU dial while its request queue backs up for miles. Scale on queue depth instead.
4 min
Any-LLM vs LiteLLM: You're Comparing a Library to a Building
Mozilla's any-llm and LiteLLM get pitted against each other constantly, but they answer different questions — the only one that matters is whether you actually need a proxy.
5 min
Request Hedging for LLM Tail Latency: Race the Slow Call, Don't Retry It
Every other latency fix speeds up the typical request. Hedging is the only one that attacks the slow tail — by firing a duplicate after your p95 and taking whichever finishes first.
5 min
How to Set a Timeout for an AI Agent: A Per-Call Timeout Won't Bound the Loop
The SDK's 10-minute default times out one call; an agent makes dozens. You need a deadline the whole loop shares — and cancelling to enforce it still costs tokens and can corrupt state.
6 min
How to Load-Test an LLM App: You're Stress-Testing the Rate Limiter, Not the Model
For an app built on a hosted LLM API, the wall you hit under load isn't the model's speed — it's the provider's rate limiter and your own retry policy. Test for the ceiling and the fall, not the throughput.
5 min
Disaggregated LLM Inference: Why Prefill and Decode Are Moving to Separate GPUs
The two halves of every LLM request fight each other on the same GPU. Disaggregated serving splits them onto separate hardware — and the win is real, but only past a certain scale.
4 min
AWS Trainium vs NVIDIA GPU for LLM Inference: The Bill Is Cheaper, the Onramp Isn't
Trainium2 and Inferentia2 sell real price-performance and AWS capacity. NVIDIA sells CUDA. The decision is whether the Neuron SDK supports your model and serving stack — and how much engineering you'll spend finding out.
5 min
KV Cache Eviction: StreamingLLM vs H2O vs SnapKV vs Quest
Three of these throw tokens away to save memory. One keeps them all and just reads less — and for a long-running agent that revisits its own past, that difference is the whole game.
5 min
NVIDIA NIM vs vLLM vs TGI: How to Self-Host LLM Inference in 2026
One of these isn't an inference engine at all — it's a wrapper around the other two. Sorting that out is the whole decision, and it just got simpler because one contender quietly left the race.
4 min
How Many GPUs to Serve an LLM: Capacity Planning Is a Memory Problem, Not a FLOPs One
Decode is memory-bandwidth bound, so a GPU's TFLOPs barely predict serving capacity. What caps concurrency is the KV cache. Here's the actual arithmetic, with a worked example.
5 min
How to Track LLM Costs Per Customer in a Multi-Tenant App
The provider's per-user field won't give you an invoice, and raw token counts lie. The honest unit of attribution is the priced token — after caching, batching, and hidden thinking.
4 min
FlashAttention vs PagedAttention: Two Different Bottlenecks, Not Two Choices
One speeds up the attention math; the other stops your KV cache from wasting most of the GPU. You run both — and the friction where they meet is the actual story.
5 min
Expert Parallelism: How Giant MoE Models Are Actually Served
A trillion-parameter MoE only fires a fraction of itself per token. Expert parallelism scatters those experts across dozens of GPUs — but the hard part was never the split. It's the all-to-all traffic and the hot experts, and they only pay off when you're drowning in load.
5 min
A Circuit Breaker for LLM API Calls — and Why It Has to Trip on Cost, Not Just Errors
The textbook breaker opens when calls start failing. The incident that actually bankrupts an agent is a loop where every call succeeds — so you need a second breaker that watches money, not errors.
4 min
AMD MI300X vs NVIDIA H100 for LLM Inference: The Memory Wall and the Software Tax
It isn't a FLOPS race. Decode is memory-bound, and the MI300X's 192 GB lets a model live on fewer GPUs than an 80 GB H100 can. The catch was never the silicon — it was ROCm. Here's where that tax stands in 2026.
5 min
Scale to Zero for LLM Inference: Why Cold Starts Are a Weight-Loading Problem
The cost of scaling a self-hosted model to zero isn't compute or container boot — it's the seconds-to-minutes of shoving tens of gigabytes of weights into empty GPU memory. That's the number that decides warm-vs-zero.
5 min
Backpressure for AI Agents: Why Exponential Backoff Makes Fan-Out Worse
When an orchestrator spawns twenty sub-agents that each retry on 429, the retries compound into a self-inflicted DDoS. The fix is upstream flow control, not smarter backoff.
5 min
OpenRouter vs LiteLLM: Which LLM Gateway for Your AI Agent Stack?
They get filed as rivals because both promise "one API for every model." But one is a hosted marketplace you buy from, the other is infrastructure you run — and the smart move is often to use both.
5 min
KV Cache Offloading: LMCache vs Mooncake vs NVIDIA Dynamo
Your engine computes a KV cache, uses it once, and throws it away. Offloading turns that scratchpad into a shared storage tier — and changes the question you should be asking.
4 min
B200 vs H200 vs H100 for LLM Inference: Pick by Memory Wall, Not Peak FLOPS
The B200's headline 5-6x throughput jump is two different upgrades wearing one number — bigger HBM and FP4 compute — and which one matters depends entirely on whether your workload is memory-bound or compute-bound.
5 min
Self-Hosting LLM Inference vs an API: The Break-Even Math
Is it cheaper to run an open model on your own GPUs than to call an API? The deciding number isn't the token price — it's how busy the GPU stays.
5 min
How to Reduce AI Agent Token Costs
The cheaper-model reflex is the wrong first move. An agent's bill is dominated by the transcript it re-sends on every step — so the money is in the context, not the price card.
5 min
How to Reduce AI Agent Latency
Buying a faster model is the reflex, and usually the wrong first move. An agent's wait is a chain of serial round-trips — so the latency is in the loop, not the tokens-per-second.
4 min
How to Handle LLM Rate Limits: Retries, Backoff, and Fallbacks Without Burning Your Bill
Every agent in production eventually meets a 429. The naive fix — just retry — is also the most expensive bug in modern LLM apps. Here's the layered pattern that survives the limit instead of paying triple for it.
5 min
How to Handle LLM API Failures: Retries, Timeouts, and Fallback Chains
A 429 means wait; a 400 means stop; a 200 from your backup model can be the most dangerous answer of all. The reliability layer every agent needs and most skip.
5 min
Why LLM Inference Isn't Deterministic — Even at Temperature 0
Greedy decoding should give the same answer every time. It doesn't — and the usual 'floating-point' excuse is wrong. The real culprit is what else is in the batch with you.
5 min
tiktoken vs SentencePiece vs Hugging Face Tokenizers
Three libraries everyone compares as if you get to choose. You don't — your model already chose for you. The real question is what that choice costs, and who pays it.
5 min
Temperature vs Top-p vs Top-k: How LLM Sampling Actually Works
Three of these knobs do the same job — truncate the unreliable tail of the next-token distribution. The differences are smaller, and more contested, than the tutorials admit. And if you build agents, you probably want almost none of it.
5 min
Intent Routing for AI Agents: When a Cosine Match Beats an LLM Call
If your agent has a fixed set of tools and intents, you probably don't need a model to pick between them. An embedding lookup is faster, cheaper, and the same input lands the same way every time.
5 min
How to Extend an LLM's Context Window: Position Interpolation vs NTK vs YaRN
Stretching a model past its trained context length isn't a memory problem — it's a positional-encoding generalization problem. The methods that work all interpolate instead of extrapolate, and the good ones interpolate unevenly.
5 min
NVIDIA Dynamo vs llm-d vs vLLM: How to Serve LLMs at Scale in 2026
"Dynamo vs vLLM" is a category error. One is an orchestrator across pools of GPUs; the other is the engine inside a single replica. Sort that out and the real choice gets clear.
5 min
MIG vs MPS vs Time-Slicing: How to Share a GPU for LLM Inference (and When Not To)
Three ways to put more than one workload on one accelerator — and a reason most LLM serving shouldn't use any of them. Choose by failure domain, not utilization.
5 min
MHA vs MQA vs GQA vs MLA: How Attention Stopped Eating Your KV Cache
Every attention variant since 2019 has been one argument about the same scarce resource — the key-value cache — and the newest answer changes the terms of the deal.
5 min
Mamba vs Transformer: Do State-Space Models Matter for Agents Yet?
Pure Mamba never beat the Transformer outright — but a wave of hybrids that keep ~8% of layers as attention now cut long-context memory 70%+ and triple decode throughput.
5 min
LLM Inference Latency: TTFT vs TPOT vs Throughput, and Why 'Tokens Per Second' Is Two Numbers
The three numbers everyone quotes measure three different bottlenecks — and per-user speed and system throughput move in opposite directions, so a vendor's headline tok/s can mean whatever flatters it.
5 min
FlashAttention vs PagedAttention vs FlashInfer: Three Different Problems, One Word
Stop choosing between them. FlashAttention is the compute kernel, PagedAttention is the memory layout, FlashInfer is the engine — a modern stack runs all three at once.
5 min
Diffusion LLMs vs Autoregressive: Why 'Parallel Generation' Wasn't Actually Faster
Diffusion language models generate every token at once instead of left-to-right, which sounds like a guaranteed speedup. The early open models were slower than the autoregressive baseline anyway — and the reason they finally got fast is the opposite of what the pitch implied.
6 min
Continuous Batching vs Static Batching: Why LLM Serving Throughput Jumps an Order of Magnitude
Static batching wastes the GPU because LLM outputs are variable-length — short replies idle while the batch waits for the longest. Continuous batching schedules at every token step instead. The catch is that the same trick that wins throughput can spike latency.
4 min
Groq vs Cerebras vs SambaNova: The Race for Faster-Than-GPU Inference
Three startups built custom silicon to outrun the GPU on token generation. The speed is real, the SRAM is tiny, and that tradeoff decides everything.
5 min
Tensor Parallelism vs Pipeline Parallelism: How to Split an LLM Across GPUs
When one model won't fit on one GPU, you have two ways to cut it up — and the right cut is a description of your interconnect, not a tuning knob you guess at.
4 min
Why LLM Inference Has Two Speeds: Continuous Batching and Prefill/Decode Disaggregation
A single tokens-per-second number hides two workloads pulling in opposite directions — and the whole arc of serving optimization is the field admitting they should never share a GPU.
5 min
MLX vs llama.cpp: Which Engine Should Run LLMs on Apple Silicon
Ollama just ripped out llama.cpp and bolted in Apple's MLX on the Mac. The switch is a tell about where your bottleneck actually lives — and when the older engine still wins.
4 min
The Cheapest LLM Tokens Are the Patient Ones: Batch APIs vs Realtime
Every major provider sells inference at roughly half price if you can wait up to 24 hours. The discount isn't the point — the contract is, and it tells you which agent work was never realtime to begin with.
4 min
KV Cache Quantization: The Memory That Actually Caps Your LLM Throughput
You quantized the weights to 4-bit and thought memory was solved. At long context the KV cache dwarfs the weights — and it needs a different kind of quantization to shrink safely.
4 min
How Much VRAM Do You Need to Serve an LLM? A 2026 Sizing Guide
The weights are the easy part — the math you can do on a napkin. What silently OOMs your server in production is the KV cache, and almost nobody budgets for it.
5 min
vLLM vs TensorRT-LLM vs TGI: Choosing a Production LLM Serving Engine
Three engines, one job: turn a model into a high-throughput endpoint. The feature gaps are closing — what's left is portability, vendor lock-in, and which project is still being built.
5 min
Speculative Decoding, Explained: Why EAGLE Beats Medusa for Faster LLM Inference
Speculative decoding makes a single LLM response 2–6x faster without changing a token of the output. The reason it works — and why the newest method wins — is a fact about your GPU, not your model.
5 min
GPU for LLM Inference: H100 vs H200 vs A100 vs L40S
Buyers shop for these cards by peak FLOPS. Token generation barely uses them. The spec that actually moves inference throughput is the one most spec sheets bury — and a single NVIDIA card proves it.
5 min
BentoML vs Ray Serve vs KServe: Choosing a Model-Serving Framework
Three ways to put a model behind an endpoint — and they increasingly run the same engine underneath, so the thing you are actually choosing is not speed.
5 min
Ollama vs LM Studio vs Jan: Running LLMs Locally in 2026
They all wrap roughly the same inference engine, so they all run the same model at roughly the same speed. The thing that actually separates them is what shape they want to be — a daemon, a polished app, or an open one.
4 min
Groq vs Together vs Fireworks: Choosing a Serverless Inference API for Open Models
Three ways to rent open-weight inference without owning a GPU — and why the fastest of them just licensed its speed to Nvidia instead of competing with it.
4 min
RouteLLM vs NotDiamond vs Martian: Do LLM Model Routers Actually Cut Costs?
Per-prompt model routing promises GPT-quality answers at a fraction of the bill. The honest 2026 answer is that it's a cost lever with a threshold, not a free one — and a neutral benchmark disagrees with the marketing.
5 min
LiteLLM vs Portkey vs TensorZero: Choosing an LLM Gateway in 2026
Every agent ends up talking to more than one model provider. The library you put in the middle decides whether that seam stays a proxy or quietly becomes your control plane.
4 min
vLLM vs SGLang vs Ollama: How to Choose an LLM Inference Engine in 2026
The benchmark everyone argues over is the wrong one. The engine you should run is decided by how much context your requests share — not by whose tokens-per-second screenshot is biggest.
5 minLatest in Inference & Gateways
Not buyer's guides — the news, teardowns, and explainers behind this topic.
