JevAgentHarness：小模型按置信度接管 agent 决策

Sumanth @Sumanth_077原推发布于 2026.10.25

在 X 上打开原推

Jev as a decision layer for LLM agents! LLM agents make a lot of small decisions between the bigger reasoning steps: which tool to call, which argument to use, whether the task is done, or what should happen next. JevAgentHarness explores what happens when those bounded decisions are shared with a much smaller decision model instead of sending every step back to the LLM. The decision model returns probabilities over the available options. If its confidence is above a threshold learned from the agent’s own trajectories, the harness uses that decision directly. If confidence is too low, control goes back to the LLM. But the paper found that standalone decision accuracy is not enough. The model that performed best on isolated decision questions completed only 69.5% of full agent tasks. And when tool choices were delegated without fitting the confidence threshold, task success dropped by 19 points. The reason is that agent decisions are sequential. One wrong tool call changes the state the agent sees next, so later decisions happen in situations the decision model may never have seen before. To address that, the researchers trained AgentDecision-4B on the states agents actually reach, including states that appear after earlier mistakes, with Jev providing the decision targets. That changes the harness from simply replacing LLM calls with a smaller model into a confidence-based control system where both models handle the decisions they are better suited for. On tasks similar to its training distribution, the resulting 4B decision models reduced agent cost by 81.1–89% and made execution up to 2.5× faster without losing 3 or more points of task success. And on shifted tasks, AgentDecision-4B reached 91.5% task success compared with 82% using Jev. The broader takeaway is useful: if you want decision models inside agents, benchmark accuracy alone is not enough. You need to evaluate them inside the actual agent loop, on the states their own decisions create. The harness, benchmark, training data, and models are open source. I’ve shared the paper in the comments!

这条是引用着另一条写的：x.com/sumanth_077/status/2101639788961112279