Five-model bug hunt: Sonnet 5.5 beats GPT-6 Astra
Across two real repos with 105 known bugs, Sonnet 5.5 at max effort found the most at 55.5 on average, ahead of GPT-6 Astra's 45 and Opus 5.5's 41.7, but it used about three times Opus's turns.
.@PawelHuryn gave five models two real repos with 105 known bugs. Sonnet 5.5 at max found the most: 55.5 on average. GPT-6 Astra: 45. Opus 5.5: 41.7. the cheaper model beat the flagship at finding bugs. the price of that: 1,330 turns, about three times what Opus used. drop Sonnet to xhigh and its turns fall by more than half. its score falls to 39. so the most thorough setting and the cheapest setting are the same model on two different dials. want every bug before a release? Sonnet at max, and budget for the turns. want a fast pass on every PR? lower effort, and accept you'll miss some. one builder's test on his repos, not a benchmark. worth knowing before you set your review agent to max.
This post quotes another post: x.com/neil_xbt/status/2105292880797483484