路由实验质疑 Jev 到底加了什么
13 模型 5835 条查询的独立路由实验里,Jev+检索 62.4%,去掉 Jev 检索单独也是 62.4%,作者质疑 Jev 在路由决策中究竟贡献了什么。
Found something about Jev that surprised me: in an independent routing experiment across 13 models and 5,835 held-out queries, Jev + retrieval reached 62.4% accuracy, beating the best single model at 60.3%. But when they removed Jev, retrieval alone still scored 62.4%. The routing gain wasn’t coming from predicting whether a task was easy or hard. It came from knowing which models had historically performed well on similar tasks. Even crazier, an oracle choosing the cheapest model that could correctly answer each query would reach 82.6% accuracy at $4.83 per 1,000 queries, compared with 62.4% at $26.73 for the tested router. So if removing Jev produced the exact same 62.4%, what exactly is Jev adding to the routing decision?