Astra solves Erdős problems but still misjudges furniture assembly
The author notes GPT-6 Astra still errs about 20% of the time on IKEA assembly judgments, arguing AI progress is overstated by tasks with clear outcomes and that AGI claims are exaggerated.
So basically, GPT-6 Astra, the most advanced AI model (the one that solves Erdos problems, one-shots 3D video games, and simulates physics), still fails to tell if you assembled your furniture correctly 20% of the time. (The other ones are even worse.) What does this mean? We over-index on problems that have very clear outcomes (like coding and math). And even the IKEA benchmark has a very clear and measurable result. Wait till it becomes saturated within the next cycle of AI model releases. What about all the small decisions we make every day, the ones that have no predictable outcome, the ones that we do based on imperfect priors and correct as we gather more information? Measuring AI progress on these kinds of tasks that are not clear-cut is hard. And that is the essence of human and general intelligence. So all the "AGI is here" claims are a bit overstated.
This post quotes another post: “Epoch AI launches FAB, a furniture assembly error-spotting benchmark”