ARC-AGI-3 实测：Astra 出现后作者放弃继续测

Psyho @FakePsyho原推发布于 2026.10.14

在 X 上打开原推

a few months ago I was extensively testing codex/claude code on the public demo of ARC-AGI-3; then astra came, obliterated the benchmark and I lost the interest originally, I started testing it, because I was tired of the misleading "Humans score 100%, AI <1%" narrative from the ARC Prize team; it also completely failed at tracking the pace of the AI progress I've never shared the results of those tests so here you go what I tested: - openai models with codex or ant models with cc - agents were sandboxed, had no internet access, all games were independent - agents were free to use the tools (e.g. create code) - prompt is a variation of PRO-LONG - agents could batch multiple commands per call; this drastically speeds up testing and reduces the costs; this has a small downside that the agent has in practice fewer turns to "think" - I've used the same scoring (RHAE) and the same action limit (5x of human baseline) as the official ARC-AGI-3 does some caveats: - runs were performed at various moments, using different versions of codex/cc; this means that quite often, used agent harness might be much more recent than the model - at least since Opus 5 / Astra 6, models generally have a very good understanding of ARC-AGI-3 env; - variance is high, but the relative order of models should be more or less stable - remember that this is the public demo and the verified benchmark uses private games; that being said, those two should be very close and in general I saw no signs that models have memorized the public games graphs: #1: RHAE vs release date; you can see the steady improvement of the models here; ARC-AGI-3 was released soon after gpt 5.4 #2: RHAE vs cost (per game); official testing had astronomical costs (20-100x higher than mine); honestly I don't know how they generated such costs - batching & code execution doesn't fully explain the difference #3: RHAE vs official RHAE; both values are on the public demo; you can see how little correlation there is between ARC Prize official "minimalistic harness" and a realistic use with a regular coding harness (e.g. codex/cc) one thing maybe worth mentioning is that I had a ton of additional runs testing variations of prompts, context limits, reasoning level and few other properties; pretty much all of those things made no significant difference on the results imho the only thing worth mentioning was that apparently a big part of the generated code was used for parsing the visual input from games; I experimented a bit with creating a special limited language for parsing the visual input and when agents were limited to using it, they were nearly on par with runs that had full access to code generation unfortunately some notable models are missing (fable, opus 4.6 & 4.7), but I've moved on to testing other things so I'm definitely not revisiting this benchmark