How to write verifiers for RL data tasks
A look at verifier design for RL tasks: deterministic checks, LLM grading, and using Jev as a judge (JaaJ) for classifier-based scoring, plus pitfalls like overfitting to difficulty ranges and poorly written verifiers.
RL data task gen 101: verifiers a few caveats and things i've noticed from poor verifier generation: > because difficulty comes in a range of 0.2-0.6 normally for frontier labs, people overfit their tasks to hit this range. this is bad because it incentivizes tasks that have the verifiers shaped around the grade, rather than the grade shaped around good verifiers > i think verifiers and the surrounding envs are the hardest piece to create within a task at the highest of levels, a verifier is the way you approach verifying that a model did the task correctly. they can be: > deterministic: unit tests, tool call checks, substring checks, actually running the code, etc > LLMaaJ: taking unstructured text and grading it, running "taste" checks on code, correct or not, scores from a rubric, having an LLM reason over a trajectory etc > most recently, JaaJ (Jev as a judge): using a classifier to grade answers shapes can look completely different across different task types. rewards can be positive or negative, to determine if you're rewarding a model for doing the correct action, or penalizing it for doing the wrong action. verifiers can be gates that automatically fail a task or remove only a portion of the score. tasks in reward can be binary 1's and 0's or gradient from 0 to 1. all of these are good ways to approach task creation, depending on the task at hand. the biggest pitfalls in verifier creation: > incorrect verifiers: this seems obvious, but often times a verifier might seem correct to you, but can be wrong for a host of reasons. if you include web search or have an extremely large environment, maybe the model found contradictory information to what you thought was the correct answer, and is still technically correct > lazy verifiers: you only check for a specific path, and so any path that is correct but outside of the scope of your verifiers is penalized > missing verifiers: you're only verifying the correct actions were taken, not punishing incorrect actions too (allows a bunch of erroneous calls, for ex: if i want to email your boss for a raise, but instead it emails your boss, your customer, and your friends, but you only grade for (1), it'd be a 1.0) what i would be focusing on as a data vendor: (1) figuring out how to make judgments for non-verifiable domains in RL tasks: things like auto-research, physics, real-world sims, and philosophy >>case in point models are still pretty bad at deep thinking & writing (2) making LLMaaJ more stable >> i think we rely too heavily on LLMaaJ assuming models are very, very smart. at high contexts / during search, etc LLMs can miss obvious stuff, so you want to confirm stability via tons of regrades and helping the model as much as possible, etc