Open-weights coding benchmark: V4.1 1017 strongest but inconsistent, much cheaper
Across four open-weights releases on one coding benchmark, DeepSeek V4.1 1017 came out strongest but inconsistent, not clearly better than the prior 4.1 on average and worse in some environments, though far cheaper to run.
We ran a batch of 4 recent open weights releases through our open-ended coding benchmark over the weekend: - Muse Spark Open: currently the strongest US open weights model. Roughly equivalent to GLM 5.3 Flash. - Beam 1.0 501B: Solid release for a new entrant, roughly competitive with DeepSeek v4 Flash (07/31). - Nemotron 3.5 Super VL: A significant leap for NVIDIA (their strongest model to date), but still not a contender. - DeepSeek v4.1 10/17: strongest of the batch, but inconsistent in our testing. On average, it is not significantly better the previous 4.1 checkpoint. It performed worse than its predecessor in some of the same environments, but its current pricing made it much cheaper to run the same benchmark. The US models are all solid releases in their respective size classes, but none are Pareto optimal, with GLM 5.3 Flash performing the same or better at lower effective price.