Same DeepSeek V4.1 Flash weights score 8.6 points apart across 9 harnesses
Across 89 runs of identical DeepSeek V4.1 Flash 1017 weights in 9 harnesses, scores ranged from 31.8 (Kilo Code) to 40.4 (pi, grok build cli), with Claude Code at 36.4 — the gap comes from the wrapper.
holy sh*t, the same model scored 31.8 in one tool and 40.4 in another everyone is asking if opus 5.5 got nerfed. a 754k-view post says yes, another one reran a single prompt 20 days apart and posted the difference before you trust a screenshot, look at this test > one model: deepseek v4.1 flash 1017 > 9 harnesses, 5 attempts each, 89 runs in total > pi harness and grok build cli: 40.4 > claude code: 36.4 > kilo code: 31.8 same weights, 8.6 points apart. that gap came from the wrapper, not the model the vendor's own harness added 2-3 points and cost about 80% more, because it spawns more subagents and writes more output what actually got me is how most of us test. one prompt, one run, one screenshot, then "nerfed" how to check if your model really got worse > pin the exact model version > keep the harness, effort and tools identical > run the same 10 tasks, 5 attempts each > compare the spread, not a single output > log tokens per task. if your memory files and tools grew, usage grows with them if the numbers still move with everything pinned, then you have a story save this before the next "nerfed" thread which harness are you running right now? full stack in the article below
This post quotes another post: x.com/kiruwaaaaaa/status/2052447866208047296