AutomationBench 上 Astra 输给 Muse，但基准不可比

Xinya Du @Xinya16原推发布于 2026.09.04

在 X 上打开原推

AutomationBench is the only overlapping benchmark in the Muse Spark 1.3 and GPT-6 Astra release tables: Muse: 49.4% Astra: 41.4% Looks like Muse wins—but the shared GPT-5.6 Sol baseline scores 46.7% in Meta’s eval vs. 18.1% in OpenAI’s. Meta specifies the public v3 task set. I couldn't find the corresponding detail for GPT-6, its evaluation is clearly much harder on the shared baseline. Same benchmark name ≠ comparable scores. Model leaderboards need eval provenance.