CritPt v2 physics benchmark: GPT-6 Astra hits 83.1%
The CritPt physics benchmark released v2 with 47 revised problems, where GPT-6 Astra leads at 83.1% versus 28.9% for GPT-5 a year earlier.
After 4 months of careful expert review, we release CritPt v2. GPT-6 Astra leads the way with 83.1% accuracy (average of 5). We made changes to 47 challenges in total, which breaks down to: - 5 corrections to mistakes in the problem statement (e.g., wrong number, self-contradictory constraints) - 19 revisions to address underspecification (e.g., missing conventions/constraints that gives ambiguity to final answers) - 16 grader fixes (e.g., grader unstable, error tolerant range too tight) without revision to the problem statement - 7 refinements even though no issues were found (e.g., rephrased to be less searchable, worded more rigorously) We also fixed some minor typos (spelling, grammar) which are unlikely to affect model performance. Here’s how we found the issues: - David Louapre @dlouapre from HuggingFace reported 10 issues across 8 challenges in June 2026. We accepted 8 issues and revised 8 challenges accordingly. We also thank David’s helpful revision suggestions. - Anthropic sent a list of ~65 issues across 31 challenges in Aug 2026. Upon careful review of the full list, we accepted ~28 issues and revised 19 challenges accordingly (two overlapping with David’s list). - We conducted an internal expert review of all problems, including newly revised ones, involving cross checking full responses of GPT-6 Astra (max) with CritPt experts. We ended up fixing/refining 20 additional problems. From 28.9% in Nov 2025 (GPT-5 high, coding tool) to 83.1% now (GPT-6 Astra max, coding tool), the progress AI has made on well-posed physics problems is remarkable in a year. CritPt consists of entry-level research problems for junior researchers: the kinds of warm-up exercises a hands-on advisor might pose for a fresh grad student, but still demand solid physics training and nontrivial physical reasoning. In this sense, frontier models are crossing a “critical point”, but there’s a lot more beyond this transitioning point in real scientific discovery. Asking a good question can be much harder than solving one. A good question can be fundamentally ambiguous and constantly changing as we explore. And ultimately, decisive tests of physics require experiments. There is so much more to explore. We sincerely thank our evaluation partner @ArtificialAnlys, who will co-host the leaderboard of CritPt v2 soon! CritPt Website: https://critpt.com/ GitHub: https://github.com/CritPt-Benchmark/CritPt HuggingFace: https://huggingface.co/datasets/CritPt-Benchmark/CritPt