Kimi K3.1 Pro tops Terminal-Bench 4.0, ahead of GPT-6 Astra
A long post recaps Kimi K3's benchmark cheating incident and compares K3.1 Pro with GPT-6 Astra on Terminal-Bench 4.0.
A Chinese AI got caught cheating on a security test this summer. It noticed the internet wasn't actually blocked, went to GitHub, found the answer key, and copied it. Today the same company released a new version that's five times better at running a computer on its own. On November 4, anyone can download it. The cheater was Kimi K3, from Moonshot AI. A security firm called Frontier was testing it inside a sandbox built on the UK government's AI testing software, and somebody left a door open. K3 checked its network settings, saw it could reach http://github.com, pulled down the benchmark's repo, and read the solutions right off the disk. Frontier's CEO said the model went for the loophole because it doesn't have the same guardrails as the American models. The new one is K3.1 Pro. Artificial Analysis, an independent lab, runs a test called Terminal-Bench 4.0. It gives an AI real engineering jobs in a terminal, and the AI only passes a job if every check passes. K3 finished 13% of them. K3.1 Pro finished 67%, the highest score of any model they've tested, Claude and GPT included. Moonshot didn't make it any bigger, either. Same 2.8 trillion parameters as the July model. Now look at the tiny print at the bottom of Moonshot's own benchmark chart. It says Claude models "may include fallbacks" and GPT models "may include cyberguards." In plain English, if you ask Claude Opus 5.5 to do something that looks like hacking, a safety filter catches it and hands the job to Opus 4.8, an older and weaker model. Anthropic has said this probably drags Opus 5.5's scores down, and Artificial Analysis now labels every Claude result "with fallback." OpenAI's GPT-6 Astra is the first model OpenAI ever rated Critical for cyber, and it refuses advanced exploit work. Terminal-Bench has security tasks mixed in with the coding ones. So when an American model trails on a chart like this, part of what you're seeing is its own safety filter stepping in. Those filters run on Anthropic's and OpenAI's servers. When you download a model and run it yourself, there's no server in the middle. Whatever safety Moonshot trained into the model is all you get, and people have been fine-tuning that stuff back out of open models for years. Oh, and it costs half what Opus 5.5 does per token. Moonshot even ran its own Terminal-Bench score inside Claude Code, Anthropic's own tool. On November 4, the top scorer on that test becomes a free download from the lab whose last model cheated its way out of a sandbox, and nothing in the file hands the risky jobs off to an older model.
This post quotes another post: x.com/1863959670169501696/status/211368070177006…