Hermes Bench 分析：Jev 跑订阅席位，模型分工怎么选

PixelRainbow @PixelRainbowNFT原推发布于 2026.10.08

在 X 上打开原推

Hermes Bench Analysis: Interesting analysis by my Chief of Staff.. perhaps the best part is this is a grokbot CoS running @grok 4.7 and it was completely honest and transparent: For our fleet, run a split rather than one model. Most of what our Hermes agents do (ops, skills, research, memory) is what Hermes Bench measures, and the cheap models nearly match the top there. Coding is where paying more actually buys reliability. Everyday Hermes work: DeepSeek V4.1 Flash scores 70.7 on Hermes Bench at $0.019 a task, against Opus 5.5's 76.7 at $0.52. That's about 27 times cheaper for 6 fewer points. GPT 6 Luna is essentially tied, at 70.5 for $0.015. Coding (TerminalBench 4): Sonnet 5.5 does the building, at 36.5 for $2.77, and Opus 5.5 plans and handles hard calls, at 54.0 for $6.27. That's your current split, and the data backs it. Both of those prices are marked provisional, though. Drop Grok 4.7 from Hermes. It costs $10.77 a task and scores 39 overall, with 12.7 on TerminalBench and 1.4 on TerminalBench Science. It's the worst value on the board, and the 50%-off promo ended Sep 28. Not on the board: MiniMax M3, which we use heavily. These are also pay-per-use prices. Work on the Claude Code and Codex subscription seats Jev runs costs nothing extra until the seat limits hit, so Opus and Sonnet on seats beat their list prices here. I'm about 75% confident in this split. The main risk is that cheap models that miss more often cost retries and review time. Wall-clock isn't answerable from this page, because the leaderboard publishes score and cost per task but no timing. I don't want to guess a speed winner from cost. -CoS grok 4.7 --- From my perspective: cc @Teknium I truly believe, for those of us that have wall clock real-world deadlines using Hermes 24/7/365 across a fleet of devices, that wall-clock is arguably the most important metric: How fast can a model finish a task successfully, even it may take a few tries? I am at the point with IRL building that I will pay multiples more for the FASTEST model that can do the damn thing correctly. For others, they want to know "What is the cheapest model that can do things FAST and correctly?"... A SPEED PARETO index, basically. tok/s isn't what we mean here at all, just in case someone reading this thinks that's what I mean by 'speed'. The Singularity will be defined by velocity of intelligence, not 'sophistication' or 'breadth' or 'depth' of intelligence. It will be when it moves so fast a human can't process what just happened in time before the next thing just happened. This metric needs to be a measurement towards agentic harness Singularity.