基准测试：Jev 先选工具、LLM 只填参数，错选率下降
100 工具 6 任务 8 模型的对比测试，Jev-classifier 模式先挑工具再让 LLM 填参数，减少 flights_hold 与 flights_book 这类干扰项导致的错误调用。
someone built a benchmark to answer one question: what happens when Jev picks the tool instead of the LLM? 100 tools. 6 tasks. 8 models. 2 modes. every result logged and visualized. the setup: llm-direct mode - the LLM sees all 100 tools on every step and picks one. jev-classifier mode - Jev reads the conversation state before every model step and picks the tool. the LLM only sees the 1 tool Jev chose and fills in the arguments. the distractor problem nobody talks about: the catalog has tools like flights_hold vs flights_book, calendar_block_time vs calendar_create_event vs calendar_create_meeting. calling the wrong one counts as a wrong tool call in the report. when the LLM sees all 100 tools on every step, it picks the wrong one more often than you'd expect. Jev picks first. the LLM gets one option. wrong tool calls drop. the models tested across both modes: Gemini 3.8 Flash, DeepSeek V4.1 Flash, GPT-5.6 Luna, GPT-5.6 Sol, Opus 5, Fable 5.1, GPT-6 Astra, Qwen 3.8 the dashboard shows: steps per task, tokens and cost, efficiency ranking, call composition (on path / extra / wrong), and per-model distribution. eval-results/ is versioned. the dashboard opens with the repository's numbers without running anything.
这条是引用着另一条写的：x.com/p0lybender/status/2105657133702193458