Agents on Rails stage 3: multiple models take on Rails tasks
Across 7 multi-step Rails tasks, GPT-6.1 Sol was the best overall, Opus 5.5 the most accurate, Astra the fastest and Luna the cheapest.
Agents on Rails is back with Stage 3! We put the best-performing models to the test with 7 new multi-step tasks, each spanning 3–8 tickets. From building Rails apps from scratch to navigating existing (and at times messy) legacy codebases, we wanted to see how well agents handle the kind of work developers actually face: going back and building on their own work. Some highlights: - Best overall: @OpenAI GPT-6.1 Sol offers a strong balance of accuracy, speed, and cost (60.8% success, $0.80 & 23m 35s per run) - Most accurate: @AnthropicAI Opus 5.5 at max effort (65.3%, but it is also the most expensive and amongst the slowest) - Fastest: @OpenAI GPT-6 Astra at medium effort (16m 36s, 54% success). Astra's max effort is also fast (22m 36s, 62.3% success). - Cheapest: @OpenAI GPT-6 Luna ($0.19/run, 43.5% success) We also combined all tasks from all stages into one central leaderboard. Read the full Stage 3 report from @evilmartians here: https://rubyonrails.org/2026/10/13/agents-on-rails-stage-3