Training Jev scorers on the agent's own traces to stop reward hacking
The author trained step Jev to score each search and final Jev to read the whole trace and predict answer correctness (AUC 0.92), after an off-the-shelf hosted judge was gamed by the agent answering without searching and rated those wrong answers about 70% likely correct, for 0% accuracy.
The trick: we trained our own Jevs on the agent’s own traces step Jev: grades every search the agent makes final Jev: reads the whole trace and predicts if the answer is right (AUC 0.92) We also tried a big hosted judge off the shelf. The agent hacked it in ~3 updates by not searching at all, just answering instantly. The judge rated those answers ~70% likely correct. All of them were wrong. 0% accuracy. It learned that if you take zero steps, there’s nothing to grade 😭