Posts

Featured Post

Cost per token, not cost per watt

Image
  Copyright: Sanjay Basu There is a particular kind of electricity in a hall where people have stopped debating whether the future is arriving and started arguing, in earnest, about how to build it. That was the AI Infra Summit this year. Every theater, main stage, wrokshop rooms, I walked past was standing-room. Laptops open on knees, phones lifted to catch a slide, the quiet hum of a few thousand people who all came to solve the same problem from slightly different angles. The expo floor sprawled shoulder to shoulder. Booths two and three deep, demos running non-stop, the aisles of Hall B a slow river of badges. And the best part of an industry this large and this fast is that it still feels small in the ways that matter. I could barely cross the floor without running into someone I’ve built with, argued with, or learned from over the last three decades. I met current and old colleagues from Oracle, AWS, EMC, Dell, HP, IBM, and a dozen startups that didn’t exist eighteen months a...
Post a Comment
Read more

Whose Now Is It?

Image
  Copyright: Sanjay Basu I run into a timestamp problem most weeks, and it has nothing to do with philosophy. A training run kicks off across a few hundred GPUs. The nodes agree, more or less, on what time it is, because we keep them disciplined with PTP and a grandmaster clock, and even then they drift by tens of nanoseconds against each other. A gradient all-reduce that finishes “at the same moment” on eight nodes did not finish at the same moment on any of them. The word “simultaneous” is a convenience we grant ourselves so the log files line up. When something goes wrong at that scale, the first thing an engineer learns is to stop trusting the wall clock and start trusting the causal order. What depended on what. Which message could not have arrived before which other message left. I bring this up because I read a lovely piece this morning, from the Starts With a Bang newsletter, asking whether time really exists. It is a good question asked well, and the answer it lands on is ...
Post a Comment
Read more

Are We Full of Dr. Strangelove?

Image
  On Thanatos and the Machines We Claim Will Bury Us Copyright: Sanjay Basu Satire. This is a philosophy essay wearing a whoopee cushion. Read it for the argument, not the arraignment — no one here is being accused of anything except perhaps a little too much drama in the group chat. There is a particular pleasure in prophesying your own doom, and this week the AI industry indulged it fully. Jacob Coxon resigned from Anthropic on Tuesday and told the internet that “the people building AI earnestly believe that it could kill us all by the end of the decade.” Two colleagues — still employed, mind you — stepped forward to say he was right. One put the odds of human extinction at north of ten percent within the decade, which is either a sober risk assessment or the most confident thing anyone has said about the 2030s while still declining to sell their stock options. I want to take them seriously. I genuinely do. But I also want to introduce them to a Viennese neurologist and a wheelc...
Post a Comment
Read more

The Cheapest Millisecond

Image
  Copyright: Sanjay Basu Notes from Chicago on why inference infrastructure stopped being a compute problem I gave a keynote at The Connected World Live in Chicago on September 9th, and I enjoyed it more than I expected to. Twelve minutes is a punishing format. It is long enough to need a real argument and short enough that a single tangent costs you a fifth of it. Constraints like that are clarifying. You find out quickly which parts of your thesis you actually believe, because there is no room to carry the parts you are only fond of. Afterward I joined a fireside chat and then a panel with NVIDIA, Edgecore, and Interglobix. More on that at the end. Here is the argument I made. The question the building has to answer has changed From 2021 to 2024 we built data centers to answer one question. How large a model can we build? That question has a shape. It wants the largest coherent cluster you can power and cool. It tolerates a bad night, because you checkpoint and resume. It books c...
Post a Comment
Read more