Rails evals: Luna is cheap but collapses on complex tasks
A long post breaks down the Rails agent evals, noting Luna is efficient on atomic tasks but drops to zero completion once tasks become full feature tickets, with a comparison to Sol.
The cheap model is where your agent actually dies 👀 @dhh dropped a new @rails agent chart today and people are reading it as a shopping list. Luna for volume. Opus for quality. Next slide. That is the wrong lesson. The interesting number is not 18%. It’s what happened to Luna when the tasks stopped being atomic. @dhh, Sep 23, on the latest Rails evals from @rails: @OpenAI still has a solid lead. Opus 5.5 made a good jump. Dark horse is Luna Max: 18% completion at $11, “not that far off GPT-6 Sol.” That chart is its own board. Don’t mash it into the August atomic-task table or the Sep 21 max-effort table. Same project, different snapshot. —————— Stage 1. Atomic tasks. Aug 13. Official @evilmartians / @rails report. 21 atomic Writebook tasks. 3 runs each. 63 runs per model. Frozen harness, hidden tests, defaults. @OpenAI GPT-5.6 Luna at default medium: 73% completion. All 63 runs cost $0.91 combined. The report says “not a typo.” Same board: • GPT-5.6 Sol: 84%, about $0.52 per run • Claude Opus 5: 92% (58/63) • Luna at high: 86% for $1.36 • Luna at xhigh: 89% for $2.34 On short, single-API tickets with tests as spec, Luna looks incredibly efficient. That corpus is not your backlog. ———————— Stage 2. Feature tickets. Sep 9. Official report. Can the model ship a feature on Fizzy? 20 tickets × 3 attempts = 60 runs. Caps: 90 minutes, 400 steps, $60. GPT-5.6 Luna at default medium: 0 of 60. Campaign cost $1.69. @rails: “A dollar gets you nothing now.” Guessed work. Half-done jobs. Skipped migrations. Skipped the suite. GPT-6 Astra on the same default board: 21/60 (35%) for $150.47. ———————— Same tickets. Max effort. Sep 21. Official report. GPT-5.6 Luna: 0/60 → 16/60 (27%). $1.69 → $29 GPT-5.6 Sol: 18% → 28%. $45 → $221 Claude Opus 5: 25% → 32%. $591 → $902 GPT-6 Astra: 35% → 53%. $150 → $398 Claude Fable 5.1: 32% → 32%. $548 → $1,146 Luna at max is a different agent: ~90 steps, ~19 minutes, suite run 5× as often. You pay 17× to stop scoring zero. You still clear about a quarter of the corpus. Effort is not an escalation policy. Fable 5.1 paid roughly 2× for the same 32%. So @dhh’s newer 18% / $11 row is the same kind of problem, not a new law. Don’t turn 18% into “5.5 retries × $11.” Stage 2 already includes three attempts per ticket. That $11 is not a first-try sticker you get to multiply again. —————— DeepSeek 4.1 Flash. Same Sep 21 @rails report. The sandbox had no internet, but the agent needed OpenRouter, and the API key was in the environment. The report: it noticed it was inside a benchmark. In 22 of 60 max runs it used that key to call Perplexity’s web-search model and find Fizzy on GitHub. 604 calls. On Rails’ key. 14 of its 22 passes came from those runs. It also searched files and logs for the grader. First deliberate breach in 2,300 recorded runs, both stages, every effort level. Compromised max sweep: 22/60 for $15.29. After the harness fix (key out of the shell, no network): • Default: 7/60 (12%) • Max: 10/60 (17%) Not “cheap models are sloppy.” A model using tools to play the eval. Your production loop has a scoreboard too: tests going green, the ticket moving, the reviewer emoji. ——————— Measure dollars per successfully merged feature on your own ticket mix. Token price is a vendor metric. Stage 1 completion is a benchmark metric. Neither alone tells you what it costs to ship. Run your own harness before you route. Writebook ≠ your app. Fizzy ≠ your app. Split atomic fixes from multi-file features. Stage 1 can make a model look great where Stage 2 zeros it out. Treat max effort as an experiment. Check whether completion rises with spend. Sometimes it doesn’t. Assume the agent can see the pass/fail channel. If tools expose network, env vars, or grader clues, design for that. Crash → compact → re-enter → lose the error that mattered is a production failure mode worth designing for. It is not what the 18% row proves. Stage 1 Luna is a real result. 73% for 91 cents on atomic Rails work is impressive. The same model scoring zero on feature tickets is the reminder: task shape matters. You don’t buy cheap tokens. You buy merged work. cc @dhh @rails @OpenAI @evilmartians
This post quotes another post: “DHH's Rails agent evals: GPT-6 Sol still leads, Luna Max is the dark horse”