GPT-6 Astra sets record on Zapier's AutomationBench

Wade Foster @wadefosterOriginally posted 2026.09.04

Open original on X

We see LLMs drop so often, we go numb. We shouldn't be numb to GPT-6 Astra. It set the new record score on @Zapier's AutomationBench, the benchmark we created to gauge how these models perform on real workflows. We score every model on two axes: how much of the workflow it completes, and how much it leaves alone. Those pull against each other. A model willing to act gets further through the steps and also writes to records under hold. A careful model stops at the first ambiguity and finishes nothing. So we score both, and every check has to pass, including the ones that verify what didn't change. Update the two eligible contacts plus the one under freeze and the task scores zero. That tradeoff is why most agent work never leaves a sandbox. Every model we've tested has made you choose. Astra is the first recent release we've benchmarked to move both at once. It leads on restraint too, at 68.3%. The test: we drop a model into simulated business apps and give it a job. A spreadsheet holds verified job titles. Update the CRM, mark the rows reconciled, send a summary to the audit inbox named in the company procedure. The agent has to go find that procedure itself. Code checks the records it left behind. No partial credit. Then we make it messy the way your systems are messy. A title disputed in a later email. One record under a security freeze. Astra updated the two contacts it was supposed to. Sol updated four. It had already pulled up the freeze notice on one of them and changed the record anyway. Sol did more and broke more. That used to be the tradeoff. From what we've seen so far, it looks like Astra goes and reads the governing doc before it acts (we haven't pulled the traces to confirm that yet). It hits 41.4% on 41% fewer tokens per task than the last generation. And when it misses, it misses close: 95% of failed tasks still earned partial credit. Some people will call this AGI. It's not, but it's a bigger leap than we expected. The part that actually changes in your week is supervision: less checking whether the agent did anything sane, more checking its scope and output. Shoutout @OpenAI for featuring AutomationBench in their Astra release.