TogetherBench 加入 Astra 等前沿模型，按稳定性与成本排名

Zhuokai Zhao @zhuokaiz原推发布于 2026.09.14

在 X 上打开原推

We've added another set of frontier models to TogetherBench: GPT-6 Astra, Grok 4.6, Gemini 3.8 Flash, Claude Opus 5 and Opus 4.7. In each evaluation dimension: pass@1 / pass² — the bars. The full bar length is pass@1, the fraction of runs that fully solve the task (judge ≥ 0.85). The solid part is pass², the share of tasks solved in both runs. The hatched tail between them is instability, tasks the model solves only some of the time. Fable 5 leads both (70% / 62%); Grok 4.6 has the largest tail (59% / 44%): decent on a good day, but least repeatable. Judge — an agentic judge (Claude Opus 4.6) scores each patch against weighted task-completion goals frozen per task, so partial credit is comparable across models. Correction — how often the simulated user has to step in: # of corrections + 0.2 × # of nudges per task. Lower is better; it measures how much hand-holding a model needs, not just whether it gets there. $ / task — new column. Average model API cost per task at each vendor's public list price, computed from actual token usage (uncached input, cached input, cache writes, output + reasoning). The most cost-efficient is Muse Spark 1.3 ($3.19), then Grok 4.6 ($3.64) and Gemini 3.8 Flash ($4.15). Fortunately or unfortunately, there is no single winner across all dimensions, so the right model really depends on what you expect from it.

这条是引用着另一条写的：「SWE-Together 编程榜：Muse Spark 1.3 性价比最高」