Jev vs GPT-6 Luna calibration compared across 113 decisions
On the same 113 decisions Jev scored lower than Luna on accuracy, but keeping only confident answers raised Jev to 98% versus Luna's 82%, and Jev gave the same answer across 15 repeats.
THE HIGHER SCORE ISN'T THE PART THAT MAKES AN AGENT SAFE Someone ran the same 113 decisions through Jev and GPT-6 Luna Luna got more right. 67% vs 81%. Ask it to think a little and it hit 99% Then they only kept the answers each model was sure about Jev's accuracy jumped to 98%. Luna barely moved. 82% Same input, 15 times: Jev never changed its answer. Luna changed on 5 of 8 borderline cases That's the split nobody is screenshotting A smarter model is better at picking. A calibrated model is better at refusing If the next step spends money, sends a message, or ships code, the useful output is sometimes nothing Accuracy is a demo metric Knowing when not to act is the product
This post quotes another post: x.com/rendaichi147380/status/2105615520284455260