Grok Bot context strategy: stop feeding it all your history
The author calculates that sending 40,000 tokens of history per call adds up to 40 billion tokens over a million calls, and argues for retrieving only what the current request needs before handing it to Grok.
you're f*cking making Grok reread your entire chat to answer “where's my order?” a 12-word question doesn't need 40,000 tokens of history. Grok can research a company, write code and work through a difficult problem. feeding it every old message, tool response and document on every turn deserves a different context strategy. here's the arithmetic for 1 million calls: › 40,000 input tokens each: 40 billion input tokens. › 2,000 input tokens each: 2 billion input tokens. that's 20× less input before accounting for different standard, cached or batch rates. it's not proof of equal answer quality or a 20× reduction in your total agent bill. output, retrieval, tools and retries are extra. the useful move is deciding what Grok needs for the current request before sending it everything you've ever collected. take an incoming customer message: › “where's my order?” → retrieve the relevant order, not the entire conversation. › “does this product fit my setup?” → send the setup details and relevant documentation. › “you promised me a refund yesterday” → retrieve the original exchange and current refund policy. › “compare this with the option we discussed earlier” → retrieve that option, or ask which one they mean. your code controls what gets retrieved and which account it belongs to. Grok handles the work that needs reasoning. the original records stay available when a sum mary isn't enough. before changing your setup, collect 100 real requests and mark which information each answer actually needs. then check: › which answers lost an important detail? › which requests retrieved the wrong records? › how often did the bot need more context anyway? › what did each correctly completed task cost? 100 examples are a starting test, not proof that shorter context works for every case. keep this calculation for your own workload: monthly calls × average input tokens ÷ 1,000,000 × input price. calculate retrieval, output, tools and retries separately. compare standard, cached and batch rates where available. if a friend is building a Grok bot, send them this before they pipe the entire chat history into every call. they need to know which context the task requires, not how much context the model can hold.
This post quotes another post: x.com/kocer_eth/status/2092216029816472002