Bug-hunt benchmark FAQ: Sonnet 5.5 needs 7x turns of Astra

Paweł Huryn @PawelHurynOriginally posted 2026.10.17

Open original on X

Before more data drops, FAQ: 1. Those are real bugs frontier models missed at the beginning of 2026 2. The score for Sonnet 5.5 (max) is correct. It does this by spending 7x more turns than GPT-6 Astra (max). 3. Unplanted bugs aren't counted. Some OpenAI models report many real, irrelevant issues to max the benchmark. 4. Yes there are more models and more effort levels. You can adjust the filters and see live results here: https://bughunt.productcompass.pm/