Dan Spivey cut his teeth at the Chicago Board Options Exchange. In 2007 a New York hedge fund hired him to build a trade around one quirk.

S&P 500 futures trade in Chicago, but the shares behind them trade in New York. That’s about 712 miles as the crow flies.

Now when one moves the other follows, but for just a tiny, split second the prices sit out of whack. It’s a fleeting arbitrage opportunity, but it was there for the taking.

Whoever could spot the gap first buys the cheap side, sells the dear one and keeps the difference. Second place to that trade gets nothing.

It was quite literally an all-or-nothing play where speed was the advantage.

Spivey wrote the program, but the fastest line to spot the gap was full, so former Netscape boss Jim Barksdale paid to build a faster one.

The old best route followed railway lines and wasted about 100 miles. By early 2009 up to 125 crews were secretly cutting a straighter trench, sawing through Allegheny rock at about 100 feet a day.

The new, shorter cable cost about $300 million and opened in June 2010. It saved about three thousandths of a second.

Why pay that much for 1/50^th of a blink on an eye?

Because, as options trader Jon Najarian told Forbes,

“Anybody pinging both markets has to be on this line, or they’re dead.”

Traders paid about $10.6 million each for a five-year lease. And while we don’t know exactly how much it made, best guess is around $2.1 billion as all 200 spots on the line were paid up for.

In the markets, in industry, in a race to finish first, speed is the advantage everyone wants. And the same applies to the AI race. It’s not about memory, it’s not about intelligence, it’s not about compute… it’s all now about speed.

Inference is where the money in AI is heading. Inference is speed.

Nvidia said as much in August, calling it “the new frontier” as AI moves from training models to running agents.

So, what is inference?

It’s the AI answering you, rather than learning. When you ask a question, every answer happens in two halves.

First the chip reads your whole prompt in one gulp, which is called prefill and is heavy compute done once.

Then it writes the reply one token at a time, a token being a word or a chunk of one. That’s decode, and for every token the chip hauls the whole model back out of memory.

The speed of every answer comes down to how fast a chip can move memory.

Why does that matter now?

Because agents don’t ask one question and wait. They write code, run it, read the error and go again, so every slow token gets repeated down the chain...

That’s why the big names in AI are racing to speed up the decode, and they’re taking very different approaches. The winner will be the fastest.

Microsoft (Nasdaq: MSFT) unveiled its Maia 300 chip this week,

“Maia 300 is designed to maximize useful tokens per watt and useful tokens per dollar.”

That’s a chip built with trade-offs if you ask me. They want it fast(ish) but not too costly.

Ain’t no hedge funds using Maia!

It claims 40% more tokens per megawatt than leading GPU systems but has about 20% less memory speed than Nvidia’s Rubin.

Alphabet’s (Nasdaq: GOOGL) TPU 8i carries 384MB of SRAM, the fastest kind of memory because it sits on the chip. It still hasn’t been released into the wild though.

Amazon (Nasdaq: AMZN) is keen on speed but didn’t build a fast chip at all. It brought in Cerebras (Nasdaq: CBRS) to run decode on its cloud while its own Trainium handles prefill.

And then there’s Nvidia. Nvidia (Nasdaq: NVDA) paid $20 billion for Groq’s technology. In August its Groq 3 racks hit 3,400 tokens a second on a 100,000-token prompt, four times Cerebras’ older CS-3.

And then when OpenAI put their latest Ultrafast model on Nvidia Blackwell instead of Cerebras, it hit CBRS stock price by around 20% in a single day.

Is speed the moat?

Most money gets made serving lots of users at once, who aren’t desperate for speed, which is Microsoft’s bet. But the organizations that need raw speed (like hedge funds) are still happy to pay up big, like the traders on Barksdale’s faster cable in 2010, to get their edge.

So, is out-and-out fastest the trade? Or pretty quick, and more scale the answer?

It comes down to memory on the chip, and each Groq chip holds 500MB, so a mid-sized model needs about 64 of them passing notes.

A Cerebras chip is a whole silicon wafer with 44GB on board, so it needs far fewer hops. Its CS-4, due later this year, claims to be streets ahead of anything else on raw speed.

And to clear up any confusion on OpenAI’s relationship with Cerebras, Sam Altman posted last week,

“Cerebras is a close partner, and we have a deep engagement pushing on the frontiers of speed.”

At about $36 billion, Cerebras is worth roughly 1/150^th of Nvidia...

Of course, speed today is not speed tomorrow. And ongoing increases in speed and innovation will be required. By 2012, microwave links did Spivey’s trip on Barksdale’s cable in about 8.2 milliseconds against the cable’s 13.

Nvidia is the safe answer their scale and resources will extract the most from Groq technology.

But for the fastest chips made for inference, and the one with far more asymmetric upside it’s hard to look past Cerebras.

More soon,

Lewis Maguire