Memory shortage risks pushing AI past expensive chips entirely

Who: Posted by @EugeneNg (equity investor, tech/AI commentator), sharing reflections on remarks made by Jonathan Ross — founder of Groq, architect of Google's first TPU — at the Sohn investment conference, where Ross is now at NVIDIA.

What's new: The argument is that memory chipmakers are caught in a self-defeating trap. By keeping supply tight to protect high profit margins, they are giving the world's best engineers a powerful incentive to make memory-hungry software leaner — and if that engineering effort succeeds at scale, it will permanently shrink demand for the memory those companies sell.

How it works: The tension pivots on Jevons Paradox. Normally, cheaper memory means engineers build bigger, more ambitious models, which in turn consumes even more memory — a self-reinforcing growth loop. But the paradox has a failure condition: if scarcity lasts long enough, engineers stop treating memory as a given and start treating it as a constraint to engineer away entirely, through smarter software, leaner model designs, or alternative hardware. At that point the loop runs in reverse.

The numbers: The concrete example offered is DeepSeek-V4-Pro, which achieves a 90 percent reduction in KV cache memory burden compared to its predecessor DeepSeek-V3.2 — a dramatic illustration that software innovation can cut hardware demand faster than chipmakers can adjust their supply strategy.

Why it matters: The strategic implication is aimed squarely at memory manufacturers like SK Hynix and Micron. Ross's point is that these companies should be racing to build more fabrication capacity and drop prices before the engineering workaround becomes permanent. A company like TSMC is cited as the counter-model: it expanded aggressively rather than rationing supply, kept engineers dependent on its product, and captured the long-term demand wave rather than sacrificing it for near-term pricing power.