The Cheapest Millisecond

 

Copyright: Sanjay Basu

Notes from Chicago on why inference infrastructure stopped being a compute problem

I gave a keynote at The Connected World Live in Chicago on September 9th, and I enjoyed it more than I expected to.

Twelve minutes is a punishing format. It is long enough to need a real argument and short enough that a single tangent costs you a fifth of it. Constraints like that are clarifying. You find out quickly which parts of your thesis you actually believe, because there is no room to carry the parts you are only fond of.

Afterward I joined a fireside chat and then a panel with NVIDIA, Edgecore, and Interglobix. More on that at the end.

Here is the argument I made.

The question the building has to answer has changed

From 2021 to 2024 we built data centers to answer one question. How large a model can we build?

That question has a shape. It wants the largest coherent cluster you can power and cool. It tolerates a bad night, because you checkpoint and resume. It books capacity in GPU hours and measures success in time to convergence. Its traffic is a rhythm you can plan around, because synchronous all-reduce is predictable by construction.

From 2025 onward the question is different. What does the next token cost, and how far away is the person waiting for it?

That question has a completely different shape. It never ends, so there is no maintenance window. It measures utilization in milliseconds rather than hours. Its traffic is diurnal, global, bursty, and increasingly all-to-all. And it is unforgiving in a way training never was, because when it fails a customer sees it happen.

Every one of those differences is an infrastructure requirement the training era never had to satisfy.

The second question is also not answered in the chip. It is answered in the fabric, in the megawatt, and on the map.

One equation now governs the design

Cost per million tokens equals all-in cost per hour, divided by tokens delivered inside SLO per hour.

That is the whole thing. It fits on one line and it reorganizes the building.

The numerator is a facilities decision. Capital amortization per rack, power and cooling, the fabric build, optics, land, staff, spares. It is real money, and it is largely bounded, because everyone in this industry is buying from the same supply chain within a few percent of each other.

The denominator is an architecture decision. Batching and scheduling policy, where the KV cache lives and how often it is reused, model shape and sparsity and quantization, and how much headroom you hold back for the tail.

Most organizations negotiate the numerator carefully and neglect the denominator entirely. The denominator has far more range.

Note the three words doing the work in it: inside SLO. A token delivered outside the latency objective has no value and no revenue. You produced it, you paid for the power that made it, and you cannot sell it.

Which gives the sentence I care most about in this whole argument. Throughput is vanity. Goodput is the only number that pays for the building.

What that range actually looks like

Let me put numbers on it, with the disclaimer I gave from the stage. These are illustrative. I chose them to show the shape of the arithmetic, not to benchmark any platform.

Take one serving node. Eight accelerators. Thirty dollars an hour all in, covering capital, power, cooling, fabric, and facility. Identical model weights in both cases.

Well scheduled, that node sustains ten thousand output tokens per second inside a fifty millisecond inter-token objective. Ten thousand a second is thirty six million an hour. Thirty dollars divided by thirty six million tokens is eighty three cents per million.

Poorly scheduled, the same node sustains four thousand tokens per second inside the same objective. Now it is two dollars and eight cents per million.

Same silicon. Same model. Same power bill. Two and a half times the cost.

Nothing about that delta appears on a purchase order. It came from batching policy and from where the KV cache lived.

Why this becomes a networking problem

Here is the part that surprises people who have not sat with an inference profiler.

A single request is not one workload. It is two, wearing one costume.

Prefill reads the entire prompt at once. It is compute bound, parallel across the sequence, and it saturates the tensor cores. It drives your peak FLOPS and your peak power draw. It sets time to first token.

Decode emits one token at a time. It is memory bandwidth bound, serial by construction, and it cannot be parallelized away. The accelerator spends most of its time waiting on HBM. It drives interconnect and cache pressure. It sets inter-token latency.

Run them on the same node and each starves the other. You tune batching for prefill and decode suffers, or the reverse. This is why the industry is disaggregating them into separate pools, and disaggregation is the right answer.

It also hands the bill to the network. The KV cache built during prefill, which is gigabytes per request, now has to reach a different pool before decode can begin, at microsecond urgency.

Disaggregated serving converts a memory problem into a network problem.

If you serve a sparse expert model, add a second effect. Every token's route through the experts is an all-to-all event. The network is no longer around the model. It is inside it.

Three fabrics doing three different jobs

Once you accept that, the network stops being one thing and becomes three.

Scale-up, inside the rack. A coherent accelerator domain running at terabytes per second. This decides how large a model you can serve without ever leaving the rack, and therefore how much traffic never touches a switch at all.

Scale-out, across the hall. Lossless 400G and 800G fabric carrying KV cache transfers and expert routing. The design target here has quietly changed. Nobody's user experience is hurt by mediocre average bandwidth. Everybody's is hurt by a long tail.

Scale-across, between metros. The distance to the person waiting. This tier sets time to first token more decisively than any accelerator decision you will make, and you cannot upgrade your way out of it.

In the training era we tuned for the average. In the inference era we sell the 99th percentile.

The map is part of the system diagram

Consider a conversational budget of roughly three hundred milliseconds to first token, and where it actually goes. Client to region round trip. Queue and admission. A retrieval hop. Prefill compute. Safety, tokenization, first decode.

Prefill compute is about a third of it.

Chicago to Ashburn is roughly twenty milliseconds round trip. Chicago to Frankfurt is closer to ninety. That is spent before a single token has been computed.

This is not an argument that accelerators do not matter. It is an argument that you cannot buy your way out of the speed of light. If a third of your budget is already gone to distance and orchestration before compute begins, doubling accelerator performance moves what the user feels by a fraction of what the invoice implies.

So inference capacity gets sited by population density and available megawatts, not by cheap land. Placement became an architecture decision while most organizations were still treating it as a real estate decision.

Topology, not feature

I have been making a version of this argument in other contexts for a while, and Chicago is where it snapped into focus for infrastructure.

Performance in an inference system is not a property of any component. It is a property of the arrangement. Where prefill sits relative to decode. Where the cache lives relative to both. Where the metro sits relative to the user. Change none of the parts and change only the arrangement, and you get the two and a half times I showed above.

Cognition is a topology rather than a feature. So is the economics of serving it.

That is why the vendor conversation, which is almost entirely about components, systematically underdescribes the thing you are actually buying.

Five numbers to run the building by

If you take nothing else:

1. Plan in megawatts, measure in tokens. The unit of capacity is no longer the accelerator. Tokens per second per megawatt is the yield metric for the facility.
2. Report goodput, not throughput. Only tokens delivered inside the objective have a price. Require that every vendor quote be stated at a p99 latency, because a peak number at unbounded latency is a number you cannot sell.
3. Engineer the tail. The 99th percentile of inter-token latency is not a reliability statistic. It is the product.
4. Treat the KV cache as infrastructure. Cache hit rate is a line item on the P&L. Where the cache lives is a network design decision, not a framework setting.
5. Design for 24x7. Training tolerated a maintenance window. Inference does not. Fault domains and live upgrade paths are economics now, not operations.

The room afterward

The fireside chat and the panel that followed put NVIDIA, Oracle, Edgecore, and InterGlobix (moderator) on the same stage, which is a better lineup for this argument than it might look at first.

Silicon, datacenter, and interconnection are three different companies owning three different segments of one latency budget. That budget is what the customer experiences, and no single one of those three can see all of it. The gap between what any one vendor optimizes and what the end user feels is precisely the gap my equation is trying to measure.

Getting all three in one conversation is rarer than it should be. My thanks to the organizers, and to everyone who came up afterward with a harder question than the ones I had prepared for.

The close

The training era was won with the largest cluster.

The inference era will be won with the cheapest millisecond.

Token economics is not a finance topic that arrived late to an engineering conversation. It is the design constraint that decides what we build, how we connect it, and where we put it.

Fabric. Megawatt. Map.

---

Dr. Sanjay Basu leads GPU and Gen AI Solutions and Services in Cloud Engineering at Oracle. A Technocrat's Discernment appears on LinkedIn, Medium, and sanjaysays.com.