ARC-AGI-3 evaluation: Astra beats median human action count on 96% of levels
ARC Prize ran an independent ARC-AGI-3 evaluation against roughly 500 ordinary people as a baseline, finding Astra finished 96% of levels in fewer actions than the median human, with 51.7% fewer actions per level on average.
GPT-6 Astra can learn a game nobody explained to it, then finish it in fewer moves than the median human. ARC Prize ran an independent evaluation and published the full analysis. i wrote a 6-page paper on what it actually teaches, because the useful part isn't the score. first, what the test actually is: ARC-AGI-3 drops an agent into a turn-based environment and explains nothing. no rules. no goal. no instructions. it has to poke around, figure out the mechanics, work out what winning looks like, then plan a route there. humans solve 100% of these. the best frontier score before Astra was 7.78%. 1. ARC Prize tested ~500 ordinary people, not puzzle enthusiasts, and used the MEDIAN number of actions taken to finish each level as the human baseline. 2. Astra used fewer actions than that baseline on 96.0% of levels, with 51.7% fewer actions per level on average. 3. this matters more than the score. completion tells you whether the agent solved the task. action count tells you how much trial and error it needed to understand the task. those are two very different questions, and most benchmarks only measure the first. 4. now for the part that should change how you configure agents. on the Provider Adapter harness, the total bill went DOWN at every step as reasoning effort went UP. $23,457 at the bottom. $17,332 at max. monotonic. no exceptions. 5. the mechanism is simple. a model that understands the mechanics sooner takes fewer actions. fewer actions means fewer model calls and fewer tokens. you're not really paying for thinking. you're paying for flailing. 6. so the instinct to drop reasoning effort to save money can actually cost you more on long-horizon work. measure it on your own workload before assuming low effort is cheaper. 7. how Astra thinks is worth seeing too. it doesn't take notes in normal sentences. it compresses each environment into its own little algebra. "extend8 to3; retract10 to2; shorten8 to1" that's a real line from a replay. objects, coordinates, rules and ordered plans, all compressed into shorthand it invented on the spot. 8. give it a sandbox and it stops taking notes and starts writing software. on one maze game with guards and patrols, it built: maze_solver.py then combat_solver.py then patrol_solver.py then sync_state.py to check its predictions against what it actually observed. 9. and the harness matters almost as much as the model. the same Astra scores 62.7% on a minimal provider-neutral setup and 99.9% when it can keep reasoning state between calls and compact long runs. if your agent forgets between steps, you're leaving most of the model on the table. things you should know: don't assume lower reasoning effort always means lower cost. sometimes, giving Astra more time to think helps it understand the task faster. once it understands the task, it makes fewer mistakes, takes fewer actions, and uses fewer tokens. so on long tasks, more reasoning can actually cost less than letting the model figure things out through trial and error. full paper below, with every figure from ARC Prize's analysis.
This post quotes another post: x.com/mikenevermiss/status/2098270564016091458