Performance per Whose Dollar?

TPUs, agentic inference, and the end of the one-chip cloud

On November 5, 2026, SemiAnalysis published benchmark results that would have sounded strange two years ago. Google’s TPUv7, called Ironwood, matched or beat NVIDIA’s best on cost for several frontier open models. The test was not a simple chatbot prompt. It was long, multi-turn, agentic work. That is the kind of serving where inference spend is heading.

For a decade, the case for staying on NVIDIA was never only about the silicon. It was about the software. CUDA, the kernels, and thousands of maintainers who tuned for NVIDIA first. That moat is now being contested. And it is being contested by the same open source people who helped build it.

What changed in three weeks

On October 20, at PyTorch Conference 2026, Google open-sourced TorchTPU under the Apache 2.0 license. It was built with Meta’s PyTorch team. It runs PyTorch natively on TPUs. You debug in eager mode. You compile with torch.compile for production. There are no wrapper types to learn.

On the same day, Google announced TPU backends for the two most popular open inference engines. vLLM-TorchTPU was built with Inferact, a company founded by vLLM’s creators. SGLang-TorchTPU was built with RadixArk, founded by SGLang’s creators. Google says both run in production with preview customers. It says Meta runs SGLang-TorchTPU in production.

Then came the November 5 numbers.

Read in that order, this is a software story first. A chip is only as useful as the code that can reach it. That code just got much closer.

Read the numbers like a buyer

People who buy compute do not buy a headline. They buy a price, at a speed, for a model they actually serve. So let us read these results the way they would.

All figures in this section are SemiAnalysis’s claims unless I say otherwise.

At FP8, TPUv7 looks strong. On GLM-5.2, it delivers 1.2x to 2.7x the tokens per dollar of B200, B300 and MI355X, across 60 to 300 tokens per second per user.

At FP4, the picture flips. TPUv7 has no native FP4. At 100 tokens per second per user on GLM-5.3, GB300 and GB200 at FP4 deliver 3.6x and 3.5x the tokens per dollar of TPUv7. There is a small accuracy cost, because that model was not trained for FP4. FP4 on TPU waits for TPUv8, which SemiAnalysis expects in Q1 2027.

Kimi K3 shows how fast the ground moves. At launch on October 17, TPUv7 was competitive. After four NVIDIA software refreshes, GB300 at FP4 now leads by 1.5x to 2.6x. NVIDIA’s next generation, Rubin, delivers 4.3x to 5.6x TPUv7. SemiAnalysis itself calls that comparison unfair, since Rubin should face TPUv8i.

On Qwen 3.5 397B, SemiAnalysis says TPUv7 at FP8 gives 1.67x better performance per dollar than GB300 NVL72 at FP4. Google says Ironwood reached 2.01 cents per million tokens on that model, the lowest cost on the benchmark.

Then there is the cost basis. The Meta result is priced at what Meta would pay to buy TPUs. If you rent, your number is different.

So “cheapest” is never one fact. It is a model, a number format, a speed band and a price sheet. Change any one of them and the ranking can change.

Agentic inference is a memory problem

The benchmark here is called AgentX. It tries to look like real agent traffic. Sessions run for tens or hundreds of turns. Context piles up. Each turn reuses most of the one before it. Sub-agents start and stop in bursts.

Look at what won. Not raw FLOPs. The wins came from system design.

Prefill and decode ran on separate hosts. The KV cache spilled to host memory through TPU Sync, a library Google just open-sourced. One kernel ran a whole decode step. A router sent each turn to the host that already held its prefix. Google says that routing raised decode cache hit rates from 56% to 76%.

That router is llm-d, an open source Kubernetes project. With it, SemiAnalysis reports, p90 time to first token at the fastest point was 8.6 seconds on TPUv7. GB300 at FP4 took 11.1 seconds at its fastest point.

Megakernels tell the same story. AMD reports that its version cut a GLM-5.2 decode step from about 2,459 kernel launches to 210, and ran 2.31x faster at batch 1. TPU and AMD shipped megakernels first in open engines. SemiAnalysis says NVIDIA followed within a week.

Here is why this matters to anyone who runs a GPU cloud. KV cache capacity is a cloud decision. Host memory bandwidth is a cloud decision. The fabric between prefill and decode pools is a cloud decision. So is routing. A chip vendor cannot ship these in a box. A cloud can sell them as product.

There is a counterpoint, and it is a good one. TheSequence summarized an NVIDIA Research paper this week. It found that serving open models at a 256K context window costs 3.6x to 7x more per generated token than at 16K. Then it asked whether agents need that much context at all. With frequent compaction and careful file reading, accuracy broadly held. On SWE-bench Verified, peak context stayed around 7,000 to 9,000 tokens with little or no loss.

If that holds, agents may need less long context than AgentX assumes. That would cut against TPUv7’s best results. Of course, the paper comes from a GPU vendor. Everyone in this story is making an argument. That is fine. It just means we read every chart with its author in mind.

Model shape is the quiet lock-in

One detail in the SemiAnalysis piece deserves more attention than it got.

Meta’s Muse Spark Open uses an attention head dimension of 64. On a TPU, that pads 25% of the systolic array during long-context attention. GPU matrix cores handle small tiles with little penalty. So researchers on GPUs can pick shapes for model quality and pay almost nothing for it at inference time.

SemiAnalysis says labs with large TPU fleets now design attention shapes to fit TPUs. A head dimension of 256 is the sweet spot.

Think about what that means. Hardware choice now leaks into model design. The model you pick may quietly pick your hardware.

For a cloud that serves hundreds of open models from many labs, GPU tolerance for odd shapes is a real asset. It deserves a line in the pitch.

What this means for GPU clouds

When software becomes portable, compute starts to look like a commodity. The margin moves somewhere else.

First, switching costs fall both ways. A team that serves through vLLM or SGLang can move between GPU and TPU capacity with far less pain than before. Loyalty to a chip becomes loyalty to a price and a serving experience.

Second, compute is turning into a financial product. TheSequence described proposed CME futures on H100 and B200 rental rates. Each contract covers 730 GPU-hours. ICE plans futures on ORNN’s rental indices. Much of the U.S. market is still pending regulatory review. TheSequence also made a point every cloud architect knows. Two clusters with the same GPU name can perform very differently. That gap is exactly where a cloud earns its keep.

Third, benchmarks are becoming procurement currency. SemiAnalysis says TPU customers now ask Google for AgentX results before they buy. Clouds that publish reproducible results on realistic agent workloads will be easier to buy from.

Fourth, ecosystem access is a lever. SemiAnalysis argues that NVIDIA has been slow to give cluster time to open source maintainers. Google runs a free TPU contributor pool for vLLM developers. A GPU cloud that funds maintainer CI on its own fleet helps keep the open stack GPU-first.

And then the money got nervous. On November 5, the same day these results came out, the Nasdaq fell about 5.5%. TheSequence, citing CNBC, tied the drop to worries that post-midterm scrutiny of data centers, power costs and AI regulation could slow the buildout. NVIDIA fell more than 9%. TheSequence’s editorial made the point plainly. Cheaper intelligence helps customers first. Model providers and infrastructure owners still have to show how much of that value stays with them.

That is the real question for GPU clouds. Not whether tokens get cheaper. They will. The question is who keeps the difference.

What this means for builders

If you build on top of this stack, a few habits will save you pain.

Write to the serving layer. vLLM, SGLang, llm-d and Ray are now the contract. A chip’s private kernels are not.

Benchmark on your own traffic. Multi-turn agents behave nothing like single-prompt chat. Vendor curves rarely match your speed band.

Price quality, not just tokens. FP4 tokens are cheaper. They are only cheaper if the accuracy loss is fine for your task.

Plan for a moving target. TPUv8 with FP4, NVIDIA Rubin and AMD’s next parts will reshuffle every chart within a couple of quarters.

Design for more than one kind of accelerator now. It costs less than migrating later.