Routing experiment questions what Jev actually adds
In an independent routing experiment across 13 models and 5,835 queries, Jev plus retrieval reached 62.4%, but retrieval alone also scored 62.4%, leading the author to ask what Jev contributes to routing decisions.
Found something about Jev that surprised me: in an independent routing experiment across 13 models and 5,835 held-out queries, Jev + retrieval reached 62.4% accuracy, beating the best single model at 60.3%. But when they removed Jev, retrieval alone still scored 62.4%. The routing gain wasn’t coming from predicting whether a task was easy or hard. It came from knowing which models had historically performed well on similar tasks. Even crazier, an oracle choosing the cheapest model that could correctly answer each query would reach 82.6% accuracy at $4.83 per 1,000 queries, compared with 62.4% at $26.73 for the tested router. So if removing Jev produced the exact same 62.4%, what exactly is Jev adding to the routing decision?