Beam 1.0 trails DeepSeek V4.1 Flash by 3 points on HermesBench
On 27 real assistant tasks with three trials each, both cost about 1.3 cents per task; Beam's p90 turn was 22s versus DeepSeek's 87s, and DeepSeek's lead came mainly from admitting missing access.
A brand new US lab's first open weight model just came within 3 points of DeepSeek V4.1 Flash on HermesBench, at the same price per task. We tested Beam 1.0 from @reflection_ai against @deepseek_ai's V4.1 Flash, both running in the Hermes agent (medium effort) from @NousResearch, using @VerkyYi's HermesBench. 27 real assistant tasks, 3 trials each. Two things jumped out: 1. Beam's slowest turns (p90) took 22 seconds. DeepSeek's took 87. 2. On live web research and drafting, the two were a statistical tie. DeepSeek's whole lead came from one area: being honest when it doesn't have access. Priced at @baseten list rates, both come in around 1.3 cents a task. The full task by task breakdown is in the images, and Beam wins a few you might not expect. Congrats to Reflection on a really strong first release!