Hermes benchmarks Jev compaction against its own strategy with reproducible eval
Teknium ran Jev and Hermes's existing compaction strategy on the same benchmark: Jev reached 75.5% recall at 115K tokens, Hermes 78.9% at 55K, and the eval was merged into Hermes so others can reproduce it.
This is how agent features should be evaluated There was a claim that Jev could make compaction faster and better, so Teknium actually tested it against Hermes with a reproducible eval. The result: Hermes production compaction: 78.9% recall @ 55K tokens Jev: 75.5% recall @ 115K tokens And the interesting part is why. At its default threshold, Jev dropped every one of the 851 tool-call candidates in the test. At the same token budget, Jev’s ranking tied simply keeping the most recent results. Over repeated compactions, the amount it could remove kept shrinking because user and assistant text remained. So Hermes keeps its current summary + session_search approach. But they didn’t just say “ours is better.” They merged the eval into Hermes so anyone can reproduce the results and test future compaction strategies against the same benchmark.
This post quotes another post: x.com/teknium/status/2101398453578555898