Ask "Trainium or NVIDIA for inference?" and you'll usually get a spec sheet back: petaflops, HBM, bandwidth, a dollar figure per million tokens. That comparison isn't wrong, exactly. It's just answering a question that almost never decides the outcome. The thing that decides the outcome is whether the AWS Neuron SDK will compile your model and your serving stack — and how many engineer-weeks you'll spend finding out.

Start with the hardware, because it's genuinely good. An EC2 Trn2 instance carries 16 Trainium2 chips — 128 NeuronCores, 1.5 TB of HBM at 46 TB/s, and up to 20.8 dense-FP8 petaflops. Each chip holds 96 GB of HBM at 2.9 TB/s. The inference-only sibling, Inferentia2 (Inf2), runs up to 12 chips, 384 GB of accelerator memory, and 9.8 TB/s of bandwidth, and AWS pitches it as the lowest-cost generative-AI inference in EC2. These are not toy parts.

The number AWS wants you to quote#

AWS's headline claim is that Trn2 delivers 30-40% better price-performance than its own current-gen GPU instances — specifically the P5e and P5en, which are H200-class boxes. For Inf2, AWS cites 25-40% lower cost per inference versus comparable GPU instances for supported models. It also says running Llama 405B inference on Bedrock can offer 3x higher token-generation throughput than other major clouds.

Take those at face value — they're AWS's own figures, measured on AWS's own terms — and one phrase still does all the load-bearing: for supported models. The discount is real. It is also conditional on a piece of work that the spec sheet never mentions.

Trainium's price-performance is real. It's just denominated in a currency the brochure doesn't list: the engineering hours it takes to get your model through the Neuron compiler.

Why CUDA is the actual product#

NVIDIA's moat was never only the silicon. It's that CUDA runs essentially any model on day zero, with no compilation step and the deepest library of hand-tuned kernels in the industry. A new architecture drops on Hugging Face Friday night; by Saturday someone is serving it on an H100 with vLLM and no drama. That frictionlessness is the product. The chip is how it's delivered.

Neuron works differently. It is an ahead-of-time compiler: your PyTorch, JAX, or Hugging Face model gets traced and compiled into a static graph for the NeuronCore hardware before it ever serves a token. That buys efficiency, but it imposes constraints the model-architecture-fit guidelines state plainly: dynamic shapes and arbitrary control flow are not supported. Custom ops that aren't in Neuron's coverage either wait for support or get partitioned onto the CPU, which is exactly where throughput goes to die. Mainstream transformers — Llama, Mistral, Qwen — are well covered. A bleeding-edge architecture with a novel attention variant may not be, and "may not be" is a roadmap dependency you don't have on CUDA.

This is the same shape of trade we saw with AMD MI300X vs NVIDIA H100: strong silicon, a software tax. The difference is that ROCm is trying to be a CUDA drop-in, while Neuron is unapologetically its own compiled world. You don't port to Trainium so much as recompile for it.

To be fair to Neuron, the tooling has moved fast. The SDK integrates natively with PyTorch, JAX, and Hugging Face, and ships its own NxD Inference library. Crucially, vLLM V1 now runs on Neuron through the vLLM-Neuron plugin, bringing continuous batching, speculative decoding, and an OpenAI-compatible server to Trainium and Inferentia. Amazon's own Rufus assistant scaled to multi-node inference on Trainium with vLLM, and recent Neuron releases added beta support for Qwen3 MoE and Pixtral from HF checkpoints.

Read the fine print, though. vLLM V0 support was already sunset; some distributed-inference paths require AWS's Neuron fork rather than upstream vLLM; and "supported" still means a narrower set than CUDA's everything-everywhere. The well-trodden path is now smooth. Step off it and you feel the difference immediately.

The capacity card#

There's one argument for Trainium that has nothing to do with price or software: you can get the chips. Through 2024-2026, frontier GPU supply was the binding constraint for half the industry, and AWS's pitch — committed Trainium2 capacity, with Trainium3 following — is partly a pitch about availability. The marquee proof point is verifiable and not subtle: Anthropic runs and trains Claude on Project Rainier, a cluster that grew from roughly 500,000 to over one million Trainium2 chips across multiple states, with the majority of chips used for inference. Anthropic has also committed over $100 billion to AWS and secured up to 5 GW of capacity.

But read Anthropic's own framing carefully. The company says it runs Claude across "a range of AI hardware — AWS Trainium, Google TPUs, and NVIDIA GPUs" so it can "match workloads to the chips best suited for them." That is not an endorsement of Trainium as a GPU replacement. It's a statement that, at frontier scale, the smart move is a portfolio — and that the entity best positioned to absorb Neuron's tooling cost is one co-designing the stack with AWS and serving enough volume to amortize it across billions of tokens.

Who should actually consider it#

The honest verdict tracks scale and stickiness. If you're already deep in AWS, serving a Neuron-supported model at steady, high volume, and you can spend the up-front engineering to compile and tune it, the 30-40% is a real, recurring saving — Inf2 for stable inference, Trn2 if you also train. If you're chasing the newest architectures, prizing research velocity, running multi-cloud, or you can't afford a model whose support is a roadmap item, CUDA's frictionlessness is worth the premium, full stop.

And for most teams the prior question is whether to own any silicon at all — the self-hosting versus API math only favors hardware above a real utilization floor, and below it scale-to-zero serving or a managed API beats every chip on this page. Trainium's price-performance is genuine. Just remember you're not comparing two chips. You're comparing a cheaper bill against an ecosystem that asks you for nothing — and pricing the difference in your own engineers' time.