Jev vs six LLMs in a Snake experiment
Under the same seed and the same four-way decision space, Jev scored 21 points in 60 seconds, beating six models including Haiku and Opus, with a median decision time of 263 ms.
“Models have been superhuman at chat for years. So where is all the automation?” I took @CompleteSkeptic’s question literally and built a small experiment. Jev vs. six LLMs playing Snake. Same seed, same four-way decision space, 60 seconds each. Jev: 21 pts / 204 decisions Haiku: 9 / 73 Luna: 5 / 43 Sol: 5 / 40 Kimi K2.7: 5 / 34 Opus: 3 / 25 Kimi K3: 2 / 14 My first thought was: maybe Jev is simply buying speed with worse decisions. But at equal move counts: 25: Opus 3, Jev 3 40: Sol 5, Jev 5 43: Luna 5, Jev 5 73: Haiku 9, Jev 10 Then I noticed something I think is more interesting. Jev’s median decision took 263 ms. Its 16 least-confident decisions took 280 ms. Uncertainty did not really turn into more computation. It showed up as uncertainty. This made me wonder if we sometimes conflate intelligence with deliberation. A lot of software does not need a long answer. It needs a small decision, over and over again — and some indication of when that decision deserves more thought. There is potentially a very large design space here: payments, fraud, routing, logistics, agent loops — anywhere decisions arrive much faster than humans could ever inspect them. Perhaps the path from chat intelligence to large-scale automation is not making every decision think harder. Perhaps it is learning where thinking is actually worth spending. http://github.com/angelgalvisc/snake-arena-jev-vs-llms Really interesting work by @typesafeai and @CompleteSkeptic. Lots to explore here.