GPT-6 Sol and Luna regress on vulnerability detection benchmark
On the same 100 vulnerable pull requests, GPT-6 Sol found 39 and Luna 21, both below their predecessors and behind cheaper open-weight models.
We gave OpenAI’s new GPT-6 Sol and GPT-6 Luna the same 100 vulnerable pull requests as their predecessors. Both regressed. GPT-6 Sol found 39/100 for $46.92, down from GPT-5.6 Sol’s 50/100 for $281. GPT-6 Luna found just 21/100 for $2.71, down from GPT-5.6 Luna’s 38/100 for $39. The cost reductions are substantial, but open-weight models make the trade-off difficult to justify. Xiaomi mimo-v2.6-pro found 46 for $8.27, seven more than GPT-6 Sol at nearly 6× lower cost. mimo-v2.6-flash found 41 for $3.47—twenty more than GPT-6 Luna for only $0.76 extra. DeepSeek V4.1 Flash also found 44 for $12.79, beating GPT-6 Sol while costing nearly 4× less. OpenAI made this generation dramatically cheaper, but it also made it substantially worse at finding vulnerabilities. On VulnPR-100, the strongest open-weight models are now both better and cheaper.