GPT-6 Sol at max roughly matches GPT-5.6 Sol at medium on bug-hunting benchmark, test shows
PawelHuryn shares bug-hunting benchmark data: GPT-6 Sol at max effort only matches GPT-5.6 Sol at medium and trails Opus 5.5 at medium.
More data: - GTP-6 Sol (max) ~= GPT-5.6 Sol (medium) - GPT-6 Sol (max) < Opus 5.5 (medium) Live benchmarks: https://bughunt.productcompass.pm/
I finally tested GPT-6 Sol on a real work. 2 repos. 105 hidden bugs. Find and fixed what you can. It looks like a huge degradation. The results: - GPT-6 Astra (max): 45 - GPT-5.6 Sol (max): 43.5 - Opus 5.5 (max): 41.7 - Muse Spark 1.3 (max): 32.2 - GPT-6 Sol (max): 29.3 Until you measure the cost (API-equivalent): - GPT-6 Astra (max): $33.04 - GPT-5.6 Sol (max): $95.25 - Opus 5.5 (max): $58.53 - Muse Spark 1.3 (max): $18.11 - GPT-6 Sol (max): $9.93 More effort levels (xhigh, high, medium, low) dropping in this thread today 🧵
