Opus 5.5 Blender scripts: top index score, wrong library version
The author had Opus 5.5 write Blender Python for a 4.2 render farm; 7 of 11 drifted calls passed silently, showing benchmarks ignore runtime versions.
Opus 5.5 sits at the top of the Artificial Analysis index, 58 against 53 for the next two models. It still writes Blender 4.2 Python for a 5.2 render farm, and the farm never complains. This isn’t a capability gap. The index runs ten evals: office work, terminal tasks, science code, long context, hard reasoning. None of them asks which build of a library is installed on the machine that runs the code. Where the score and the scene stop agreeing: → it leads six of the ten, and not one of them pins a version to the runtime → at max effort it spends about 119k output tokens per task, all of it careful reasoning inside the API it remembers → the runtime absorbs most stale calls instead of raising, so the loop never gets an error to correct against → a coherent script for the wrong release reads as competent to anyone not diffing it against the live build → run the same file on 4.2 and 5.2 and plot the outputs against each other: the engine line sits on the diagonal, the node and pass lines fall off it In this run, 7 of 11 drifted calls went through without a word. The last log line before the next frame came from the human reviewer: looks fine to me. The leaderboard measured the thinking. The farm needed a date. You don’t need a smarter model here, you need one that’s been told which build it’s standing on. The full desk, with every call sorted by how quietly it broke, is below 👇
This post quotes another post: x.com/0xcodio/status/2114351428433011132