Command Code rerun on DeepSeek V4.1 Flash: 0 failed calls in 249 requests, 3.5–7.1s TTFT
With the same prompt and MAX effort, Command Code's harness ran V4.1 Flash and Flash Fast successfully first try with 0 failed calls across 249 requests, 3.5–7.1s average TTFT and 6–14s P95; the author open-sourced all session files and questions the 34 transport errors seen on DSH.
Harness matters. Same everything @CommandCodeAI harness had ZERO errors, check the open sourced run, session files, and how I measured it. I'd love to see if what's wrong on your end, is that your network, are you using a VPN, or is it that DeepSeek DSH harness is so bad at working with a standard API endpoint? More transparency will help. Share the session files, we can run Command on it. Same prompt, same MAX effort, run through the Command Code harness on DeepSeek V4.1 Flash and V4.1 Flash Fast: ✅ Both runs succeeded, first try: 58m 20s and 1h 29m ✅ 0 failed calls across 249 requests ✅ 3.5s to 7.1s avg TTFT. Yours: 48s ✅ 6s to 14s P95 TTFT. Yours: 154s Flash fast took longer, coz models are not deterministic, they can make and do different things with same prompts. Command Code flies on these models. DSH somehow errors out. I’ve open sourced everything: both full session files, raw event logs, every file the models wrote, and the scripts behind every number. Play both games yourself: https://commandcodeai.github.io/benchmark-deepseek-v4.1-flash-rematch/ Open sourced here: https://github.com/CommandCodeAI/benchmark-deepseek-v4.1-flash-rematch Again, no idea what’s going wrong on your end. Could be your network, a VPN, or DSH not playing well with our standard API. 34 transport errors smells like the connection issues or harness issues, not the model. So let’s find out. Open source the full session files for all your runs and I’ll run Command Code over them to see exactly what failed. If there’s anything we can improve for you, we will. Otherwise, as always: use a better harness. @CommandCodeAI recursively self-improves on every open model, and this is a nice experiment to prove that. I'm posting it within 2hrs of your post, which is how long it took to run both in parallel. This is as raw as it gets. Waiting to see your session files. Why were your runs so bad? What were the issues you faced.