Opus 5.5 reaches 40% on AutomationBench, still behind GPT-6 Astra
Opus 5.5 raises AutomationBench from 26.9% to 40.0%, while GPT-6 Astra scores 41.4%, and multi-app tasks still fail because the model acts before exploring its sources.
AutomationBench rose from 26.9% on Opus 5 to 40.0% on Opus 5.5! GPT-6 Astra scores 41.4%, so this is not a clean win. multi-app work still fails the same way. across email, documents, spreadsheets and a CRM, the model acts before it looks around. Anthropic's guide has a fix that needs no new model: one sentence in the prompt telling it to explore the relevant sources before it acts. Anthropic says that made it complete noticeably more multi-app tasks correctly. and the sentence cuts both ways. a model told to act on what it finds will act on an injected instruction too, so it only goes at sources you trust. Which of your sources would you not point it at?
This post quotes another post: x.com/neil_xbt/status/2105292880797483484