deepseek-v4.1IMAGE
tool-eval-bench: DeepSeek V4.1 Flash 1017 scores 92/100 after json_schema fix
On the official API with thinking on, fixing json_schema to json_object raised scores to 92/100 at temp 1.0 and 90/100 at temp 0, up from 87 and 85.
Ivan Fioravanti
@ivanfioravantiArchived 10 days agoOriginally posted 2026.10.17
Ivan Fioravanti @ivanfioravantiOriginally posted 2026.10.17
tool-eval-bench DeepSeek Flash v4.1 1017 on the official API (http://api.deepseek.com), thinking on. We were sending response_format json_schema; DeepSeek only supports json_object. PR created. Temp 1.0 (top_p 1.0): 92/100 (176/192) Temp 0: 90/100 (172/192) Before the fix: 87 (temp 1.0) and 85 (temp 0). Caveat: the after-fix runs are also on a newer bench build, so not every gain is the fix alone.
x.com/ivanfioravanti/status/2111398664522391796Archived 2026.10.18 16:42