Benchmark: Jev picks the tool first and the LLM only fills arguments, cutting wrong calls
A comparison of 100 tools, 6 tasks and 8 models shows that Jev-classifier mode, where Jev selects the tool before the LLM fills in arguments, reduces wrong calls caused by distractors like flights_hold versus flights_book.
someone built a benchmark to answer one question: what happens when Jev picks the tool instead of the LLM? 100 tools. 6 tasks. 8 models. 2 modes. every result logged and visualized. the setup: llm-direct mode - the LLM sees all 100 tools on every step and picks one. jev-classifier mode - Jev reads the conversation state before every model step and picks the tool. the LLM only sees the 1 tool Jev chose and fills in the arguments. the distractor problem nobody talks about: the catalog has tools like flights_hold vs flights_book, calendar_block_time vs calendar_create_event vs calendar_create_meeting. calling the wrong one counts as a wrong tool call in the report. when the LLM sees all 100 tools on every step, it picks the wrong one more often than you'd expect. Jev picks first. the LLM gets one option. wrong tool calls drop. the models tested across both modes: Gemini 3.8 Flash, DeepSeek V4.1 Flash, GPT-5.6 Luna, GPT-5.6 Sol, Opus 5, Fable 5.1, GPT-6 Astra, Qwen 3.8 the dashboard shows: steps per task, tokens and cost, efficiency ranking, call composition (on path / extra / wrong), and per-model distribution. eval-results/ is versioned. the dashboard opens with the repository's numbers without running anything.
This post quotes another post: x.com/p0lybender/status/2105657133702193458