GPT-6 Luna claims tasks are done at low reasoning effort without doing them, VulcanBench author says
The VulcanBench author says Luna at low reasoning effort declares jobs done in about three minutes without solving them correctly, with Extra High and Max results still pending.
Waking up to see how my GPT-6 Luna benchmark is going with @VulcanBench 🖖 Now that Opus 5.5 is done, GPT 6 Luna and Sol are next. So far Low, Medium, and High have run, Extra High and Max still need to run. But I thought I'd share something interesting from the first three effort levels. What seems to be happening, at lower effort levels with Luna, is, it does't take the time it needs to solve the tasks correctly. Without more thinking depth, Luna just says, "jobs done!" but it's not, and it does it in like three minutes, when it has three hours to actually do it right. More details below, will share the full results once all effort levels are completed.