TogetherBench adds Astra and other frontier models, ranking them by stability and cost
A new evaluation compares Astra, Grok 4.6, Gemini 3.8 Flash and others on pass@1/pass², correction counts and API cost per task, concluding there is no overall winner.
We've added another set of frontier models to TogetherBench: GPT-6 Astra, Grok 4.6, Gemini 3.8 Flash, Claude Opus 5 and Opus 4.7. In each evaluation dimension: pass@1 / pass² — the bars. The full bar length is pass@1, the fraction of runs that fully solve the task (judge ≥ 0.85). The solid part is pass², the share of tasks solved in both runs. The hatched tail between them is instability, tasks the model solves only some of the time. Fable 5 leads both (70% / 62%); Grok 4.6 has the largest tail (59% / 44%): decent on a good day, but least repeatable. Judge — an agentic judge (Claude Opus 4.6) scores each patch against weighted task-completion goals frozen per task, so partial credit is comparable across models. Correction — how often the simulated user has to step in: # of corrections + 0.2 × # of nudges per task. Lower is better; it measures how much hand-holding a model needs, not just whether it gets there. $ / task — new column. Average model API cost per task at each vendor's public list price, computed from actual token usage (uncached input, cached input, cache writes, output + reasoning). The most cost-efficient is Muse Spark 1.3 ($3.19), then Grok 4.6 ($3.64) and Gemini 3.8 Flash ($4.15). Fortunately or unfortunately, there is no single winner across all dimensions, so the right model really depends on what you expect from it.
This post quotes another post: “SWE-Together coding board: Muse Spark 1.3 is the best value”