Grok Bot and Muse push an HBM capacity view from bearish to neutral-to-bullish
The author was bearish on HBM capacity, but long-running always-on agents like Grok Bot and Muse keep KV caches resident and multiply concurrency, so Jevons effects eat the capacity that compression saves.
For a while I have been bearish on HBM capacity and bullish BW. Compression made capacity cheap and when v4.1 Flash took global KV down to 890B/tok and shoved the persistent cache into host DRAM and SSD I made my point again. BW is stacks x bus x width x pin rate. Height is NOT in that equation and I think base die is actually very important for BW. I still believe most of that. Everything I wrote assumed monolithic serving. You prompt, get an answer and then close out. KV lifecycle is short. Compress it 4x and you really do free up the cube buuuutttttttt agents don't close the tab how us humans would do. With the release of Grok Bot and Muse we know that the future is serving agents and now that they are more accessible to everyone we can only assume somewhat exponential growth. Context only grows with every took call, every page it reads, every screenshot, every handoff between Bots get appended. Agent trajectories are very long. Then we have to think about constant resumes. Tool returns in 2 seconds and the agent wants its cache back THIS INSTANT. And concurrency explodes like I mentioned earlier. A Grok Bot consumer is running a TEAM of bots around the clock all talking to each other. That is literally Jevons. Compression made context cheap per token so everyone burns way more tokens of context. Deepseek cut global KV bytes/tok by ~4x. Agents can EASILY multiply resident tokens by more than that. I'm not completely flipping though because BW is still king in my opinion. So I was bearish capacity but now I am neutral to slightly bullish capacity. I'll continue to expand on this