Astra trails Muse on AutomationBench, but scores aren't comparable

Xinya Du @Xinya16Originally posted 2026.09.04

Open original on X

AutomationBench is the only overlapping benchmark in the Muse Spark 1.3 and GPT-6 Astra release tables: Muse: 49.4% Astra: 41.4% Looks like Muse wins—but the shared GPT-5.6 Sol baseline scores 46.7% in Meta’s eval vs. 18.1% in OpenAI’s. Meta specifies the public v3 task set. I couldn't find the corresponding detail for GPT-6, its evaluation is clearly much harder on the shared baseline. Same benchmark name ≠ comparable scores. Model leaderboards need eval provenance.