作者回应 DeepSeek V4.1 Flash 基准测试争议：称分析方关闭图像输入

Naz @nothingto12原推发布于 2026.10.27

在 X 上打开原推

So looks like someone did an analysis (wink wink) of the session logs and posted it here: https://github.com/CommandCodeAI/benchmark-deepseek-v4.1-flash-rematch/tree/main/naz-session 👇 Let's break it down: This is a clear case-study in inherent AI bias and sycophancy. It will tell you what you want to hear and try it's best to reach a conclusion to that end. TLDR of the analysis: - It's DSH's fault - DSH's configuration was not stock (but then proceeds to say i should have used someone else's plugin to workaround issues on the backend) - TTFT numbers included failed requests My response: 1.) The analysis states that image input was enabled. Yes. But so was the sessions for OC Go and DS API. Proof in the images. Should I only use this model without image capabilities? Verifiers/QA agents use images natively to test features as a normal human would. Disabling that means I wouldn't be testing model capabilities anymore... slightly outrageous. 2.) I'm glad they ran their own tests on DSH. But the only thing it proves is that it is benchmaxxed. They disabled image inputs. The point of the benchmarks i've run is to be slightly more complex. In real workloads, you run multiple agents/subagents, and you have QA/verifiers that run on work made. The visual end result is not the point of the benchmark. It's whether complexity holds up to the task. Their run includes 0 images in all their 148 request bodies. Is that a real workflow for you? 3.) The analysis mentioned that I stopped the verifier subagent. That is correct, and here's why. The lead agent failed, and in a normal case, lead agents WAIT for subagents to finish and then they end a run. However, since the lead agent FAILED, the subagents went rogue and kept running, and failing, and running. I took the longest of the subagent runs. If I did NOT stop this already failed run, the wall-time would have been larger. Did I not end up doing a favor? 4.) Here is where I will admit I messed up, and will explicitly own up to the fault. DSH does not allow you to fork failed sessions (try yourself) thus, I had to in that session do a follow-up turn that asked for audits of TTFT. I was not clear enough to make the audit SPECIFICALLY for turn 1. So some of the audit TTFT leaked into the TTFT. Here is the corrected ones that were calculated by GPT 6.1 Sol specifically on turn 1. It's really not that different. A single extra turn will not skew results too much. 5.) TTFT measurements included failed retries/requests... yeah i'm not going to exclude them from measurements. I don't cherry pick. Methodology of the runs were 1 run 1 chance. Essentially, every other provider did fine on the tests. All the providers ran 3-4 subagents (DS API ran 4, others 3) One provider failed. However, for that one provider, it's the harness' fault. Glad the analysis was posted.

这条是引用着另一条写的：x.com/nothingto12/status/2114610137193152803