Prompt caching test across 11 providers: Gemini 4 Argon caches below documented 4,096-token minimum

Alexander Yue @Alezander9Originally posted 2026.11.05

Open original on X

Caching is a critically understudied problem Cache hit tokens are often 10% the price of a miss. I re-simulated >3B tokens of my own Mainplane sessions. Caching cuts the input bill to 15% of what it would be without it But every provider caches differently: > Who sets it: you mark the prompt (Anthropic, Qwen), or it just happens (most others) > How long: 5 min, 30 min, 1 hour, or no timer at all > How reliable: a hard timer, or a hit rate that falls every minute > How much: the whole prompt, or all but the last 2,050 tokens (Gemini) So I had a Mainplane agent swarm measure all of them. 11 provider agents, plus agents to implement and plot. >60,000 requests over 32 hours, day and night The surprise I found: many providers' official docs are just wrong 1. Meta says "You get caching with no key at all". Without a cache key, 2 of 24 repeat requests hit. With one, 26 of 27 2. Google says Gemini 4 Argon needs 4,096 tokens to cache. But a 2,661-token prompt cached 3. OpenAI says the cache lives at least 30 minutes. But multiple times in 15 hours, every cache entry we had vanished at once, and we paid for the rewrite 4. Claude Code says the cache stays warm for one TTL after the request was sent or Claude last responded. The clock starts at the send. But a read 59:55 after a response ended missed 5. Meta's 24h retention option did nothing: 0 of 30 reads hit after 30 minutes Docs drift. That's why you need real measurements Here is all the data I gathered: how long each provider really keeps your cache, with the best settings found You'd be surprised how many agent harnesses never bothered to actually measure this Mainplane now uses those settings for every provider. The code is MIT open source. Link in the comments

This post quotes another post: x.com/mainplaneai/status/2118119997176500380