ApprenticeBench: Astra and Fable 5.1 pull ahead
A new benchmark built around realistic office work shows Fable 5.1 and GPT-6 Astra clearly ahead, though cost, efficiency and natural communication still trail humans.
(1/2) ApprenticeBench probably has the strongest differentiation power of any benchmark out there. We’re seeing a step change with Fable 5.1 and GPT-6 Astra. Just look at Fable 5.1 and Opus 5: only a few points apart on AA, Terminal-Bench 4.0, and CursorBench 4.0, but 72% vs. 36% on ours! A major reason behind this is we built ApprenticeBench for ecological validity, i.e., we want it to encode all the challenges that make a job hard for a human employee to do. The agent has to learn company conventions from past bills, adapt to policy changes, and use the software to do the work. Two models can look close on any one of these challenges. However, when putting everything together, small gaps can compound into a big difference in getting the job done. No offense to open-weight models, but this gap is rough. Fable 5.1 and Kimi K3 score 53 and 44 on AA. Here, they get 72% and 18%. For the same 100 bills, it’s 28 to fix versus 82. If you’re the one doing the job, you won’t like the gap :) We also want to highlight that despite the big advantage of Fable 5.1 and GPT-6 Astra, compared with human testers, we also found non-trivial issues of those strong models in terms of cost (agents have 2.5x cost), efficiency (humans generally become faster and faster), and communication naturalness.
This post quotes another post: “ApprenticeBench: GPT-6 Astra learns on the job and beats human professionals”