Jev routing test: wrong tool loads fall from 16.8% to 7.3% but break 7 requests
In a Hermes test of 488 requests and 182 skills, a Jev suggestion lowered wrong tool loads and fixed 37 requests, but also broke 7 requests the agent had already handled correctly.
ONE JEV SUGGESTION CUT WRONG TOOL LOADS FROM 16.8% TO 7.3% THEN BROKE 7 REQUESTS THE AGENT ALREADY GOT RIGHT That is exactly the kind of benchmark AI launches usually hide TypeSafe tested 488 requests against a Hermes roster containing 182 skills Without Jev, Claude Haiku loaded the wrong skill on 16.8% of covered requests and loaded a skill unnecessarily on 9.8% of uncovered requests With a Jev suggestion, those rates fell to 7.3% and 4.0% But the router was not free intelligence It fixed 37 covered requests and damaged seven requests the agent had previously handled correctly That is the real agent-routing problem. A cheap model can shrink the tool space before an expensive model thinks. It can also confidently remove the tool that would have solved the task So routing needs a real NONE option, measured thresholds and logs showing when the suggestion changed the final choice Cheaper decisions help Unmeasured decisions just move the failure upstream
This post quotes another post: x.com/fluixoo/status/2103123493567025321