Cost per token, not cost per watt

 

Copyright: Sanjay Basu

There is a particular kind of electricity in a hall where people have stopped debating whether the future is arriving and started arguing, in earnest, about how to build it. That was the AI Infra Summit this year. Every theater, main stage, wrokshop rooms, I walked past was standing-room. Laptops open on knees, phones lifted to catch a slide, the quiet hum of a few thousand people who all came to solve the same problem from slightly different angles. The expo floor sprawled shoulder to shoulder. Booths two and three deep, demos running non-stop, the aisles of Hall B a slow river of badges. And the best part of an industry this large and this fast is that it still feels small in the ways that matter. I could barely cross the floor without running into someone I’ve built with, argued with, or learned from over the last three decades. I met current and old colleagues from Oracle, AWS, EMC, Dell, HP, IBM, and a dozen startups that didn’t exist eighteen months ago. I count it a genuine privilege to get to keep having these conversations, in person, with friends scattered across a business that reinvents itself every quarter.

Expo Theater 1, mid-session. The room stayed full clear to the aisles — the reason a summit like this is worth the trip is sitting in these chairs :-)

I had the good fortune of taking that stage in Expo Theater 1, in front of exactly the crowd you want for this argument. People who have moved past the pilot phase and are now operating what I’d call AI factories rather than demos. The moment you run a factory, the question quietly changes. It stops being how fast is the chip and becomes how many useful tokens can I ship per dollar, inside my latency budget. That is the whole talk in one line, so I’ll say it plainly here too.

Setting the frame before any hardware. The sponsor wall behind me — AWS, HCLTech, Intel, Oracle, Qualcomm — is a fair map of who is fighting over the same watt right now. My case was that the watt is the wrong thing to fight over

THE FRAME

Watts are the input. Tokens are the product.

Here is the reframe I asked the room to hold for the rest of the talk. Power is an input to the factory. Tokens are what the factory ships. If you buy and design against cost per watt, you optimize the input — and you will cheerfully purchase hardware that produces fewer tokens per dollar while congratulating yourself on efficiency. The old unit rewards buying more of the same server. It provisions to peak chip specs and optimizes the box.

The unit that actually governs the investment decision is cost per token: useful tokens delivered per dollar, inside your latency SLA. It rewards system design instead of steel. And once that is your unit, the genuinely interesting news is that software moves it more than another rack of identical GPUs does. Disaggregation, KV-cache reuse, and smart routing are where the leverage lives. Power still matters, of course — but as a constraint, not the objective.

The thesis, on the glass. Everything after this slide is the mechanism that makes the number move

WHY THIS IS HARD NOW

Two curves are bending at once

Two things are happening simultaneously, and together they are what makes this a hard problem rather than a solved one. Model sizes are growing on the order of 10× a year. We’ve gone from tens of billions of parameters into the trillion-parameter range, and most frontier open models no longer fit on a single GPU. They must be sharded. At the same time, inference itself changed character. It used to be one-shot: prompt in, answer out. Now models think, and a model that thinks generates far more tokens to reach a better answer. Token generation is climbing roughly 5× a year.

So you’re squeezed from both sides. Bigger models, and many more tokens per request. That is the pressure every architectural choice below is answering.

MECHANISM 1 · THE MODELS

MoE buys intelligence, and hands you a systems problem

Why did the frontier move to Mixture-of-Experts? Because it is a bargain. An MoE model is a large network of specialized experts, but for any given token a router picks only a few. DeepSeek R1 carries 256 experts and activates about 8 per token. You get the capacity of a very large model at a fraction of the compute and memory bandwidth. Lower cost per token. Good.

Now the bill. To serve it, you shard those experts across many GPUs; R1 across 64 GPUs is roughly four experts per device, and no single GPU holds the model. At real load, when you batch many requests together, every expert lights up and they must exchange intermediate results — heavy all-to-all traffic. MoE quietly converted inference from a single-GPU problem into a distributed-systems problem, where inter-GPU bandwidth is the thing that decides your cost. Hold that thought. Bandwidth is the constraint. It comes back.

MECHANISM 2 · THE WORKLOAD

One request. Two completely different machines.

Every inference request is really two workloads with opposite personalities. The first phase, context, or prefill, reads your entire prompt and builds the KV cache in one burst of large matrix multiplies. It is compute-bound. It wants dense FLOPS and lower parallelism, and it sets your time to first token. Then the request flips into generation, or decode. It emits tokens one at a time, each requiring a pass over the weights and the KV cache. That phase is memory-bandwidth-bound. It wants HBM bandwidth and wide parallelism, and it sets your inter-token latency.

Run both phases on the same GPUs, which is how most people serve today, and they contend. You are forced into a single configuration that is a compromise for both, and for MoE models the compromise is sharper still. Which raises the obvious question. Why are we still making them share?

Stop sharing. Split the two phases apart.

THE ANSWER

Disaggregated serving

You stop making the phases share. Prefill goes to one pool of GPUs tuned for compute; decode goes to a separate pool tuned for bandwidth. When prefill finishes, it hands the KV cache across to the decode pool, and on GB200 that handoff rides NVLink, not the data-center network. Three payoffs follow.

Optimize each phase independently

Each phase runs the parallelism and GPU count that fits it, rather than a shared compromise. Tensor parallelism for context; wide expert parallelism for generation.

Size each pool to the mix

Prefill-heavy jobs like document analysis get more context GPUs; decode-heavy chatty jobs get more generation GPUs. You match silicon to your input and output sequence lengths.

More tokens per GPU

Neither phase stalls the other, so the same silicon ships more useful tokens per dollar. This is the one that hits the cost-per-token number directly.

Disaggregation is the idea. Doing it at scale is genuinely hard, and that is where the stack earns its keep.

THE STACK · EXTREME CO-DESIGN

Dynamo does the software. GB200 NVL72 does the fabric.

Disaggregation at scale needs two things co-designed. On the software side, NVIDIA Dynamo automates it, letting the decode pool run wide expert parallelism as aggressively as it wants without ever holding back the prefill pool. On the hardware side, GB200 NVL72 puts all 72 GPUs into a single NVLink domain at 130 TB/s all-to-all. Remember that bandwidth was the constraint; this is the answer to it. The KV handoff between pools and the MoE all-to-all traffic ride the NVLink fabric, not the network.

Two numbers, kept deliberately apart. I care more about this discipline than about either figure — merging them into one hero number is exactly how vendors lose credibility.

And now the results, which I keep separate on purpose, because they are two different measurements. I won’t let them merge into a single hero number.

7× more throughput per GPU — Dynamo + GB200 NVL72 vs a single B200, DeepSeek R1 generation. SemiAnalysis InferenceXv2.

20× up to — a separate result, when disaggregation + large expert parallelism + NVFP4 + multi-token prediction stack on Blackwell. Source: NVIDIA.

2× faster inference from the KV-aware router alone. Baseten, Qwen3 480B coder, ~50K input.

4× faster time-to-first-token for agentic workloads. Llama 3.1 on Hopper, NeMo Agent Toolkit.

The point of the 20× is not the number; it is that these optimizations compound. And the point of keeping the 7× beside it, clearly labeled with its source and its workload, is that precision is the whole brand of a talk like this.

UNDER THE HOOD

What Dynamo actually is

Deploying disaggregation at scale needs orchestration at several layers, and Dynamo brings three of them under one roof. These are exactly the pieces you’d watch operate in the live demo.

KV-aware router

Unlike round-robin, it hashes the prompt prefix, tracks where cache blocks already live across the fleet, and routes each request to the replica with the best cache overlap and free capacity. That routing alone is where the 2× above comes from.

KV block manager

Cache pinning, prefetching, and eviction. Reused prefixes, a system prompt shared across many agent turns, say, resist eviction and move to host memory instead of being thrown away and recomputed. Hence the 4× TTFT on agentic work.

NIXL transfer

A low-latency data library that moves the KV cache between context and generation GPUs over any interconnect. It’s been adopted by vLLM, SGLang, and TensorRT-LLM — which tells you it is real infrastructure, not a slide.

WHERE ORACLE COMES IN

The cache is where cost per token gets won

So where does Oracle sit in all this? Right at the cache, which is precisely where cost per token is won or lost. OCI Managed KV-Cache takes attention state off the GPU and tiers it: HBM, to host memory, to NVMe, to an object tier, over NIXL. Two kinds of reuse matter. Agents that share a corpus read the prefix instead of re-prefilling it. And that reuse is cross-replica, so the cache becomes a fleet-level asset rather than being stranded on whichever GPU happened to compute it.

Why do I care so much about this one box? Because a prefill token you never recompute is the cheapest token in the building. Every prefix hit is compute you don’t pay for and latency you don’t spend. That is the economic move, stated as plainly as I know how.

And it isn’t aspirational. Dynamo 1.0 integrates with OCI managed Kubernetes today; it runs on GB200 NVL72 bare-metal GPUs on OCI Supercluster, wired with our RDMA cluster network. Production, not a lab, which, on stage, was the cue to stop talking and show it running live. Two real use cases, a Financial Services risk synthesis that is prefill-heavy, and a Life Sciences trial-protocol synthesis where eight agents share a single 520K-token corpus and the managed cache does its quiet, decisive work. Toggle disaggregation and the managed cache off and on, and you watch TTFT, inter-token latency, cache hit rate, and cost per million tokens respond in real time.

Design the data center around the token, not the watt.

THE TAKEAWAY

One line to walk out with

If you keep nothing else. Inference is two workloads, so disaggregate them. Move the KV cache over NVLink and reuse it from a managed tier, so you stop paying to recompute it. Let Dynamo do the routing and the planning; let GB200 NVL72 carry the fabric. And judge the whole thing on the one number your CFO and your latency SLA both actually care about, cost per token.

That’s the frame, that’s the stack, and the room got to see it run. My thanks to everyone who packed Expo Theater 1, to the organizers, and to the many friends old and new who made the floor feel less like a trade show and more like a reunion of people building the same strange, sprawling future together.

Screenshot from the demo