Same broken repo given to Astra and Fable 5.1: who fixes, who reviews
A controlled test found Fable made smaller, steadier changes while Astra searched wider and found more issues, leading the author to a pipeline where Astra hunts problems and Fable implements fixes.
Give Astra and Fable 5.1 the same defective repo: no bug list, no architecture brief, same starting point. Fable changed 9/9 hidden defect cases and preserved the thing it was supposed to leave alone, while Astra changed 8/9, explored much wider, and touched more than twice as many files. So Round 1 looked simple. I ended up with: Fable wins implementation. Then I swapped their work blindly where Fable reviewed Astra’s 68-file implementation and changed only 6 files and Astra reviewed Fable’s winning 30-file implementation and basically said: “You’re not done.” It reopened 30 files, added 24 tests, and took the suite from 74 → 98. I then transplanted Astra’s tests back onto untouched Fable R1: 21 tests failed. That was the moment the experiment stopped looking like Fable 5.1 vs GPT-6 Astra and started looking like an architecture. Then I tried the experiment where I gave them everything, and suddenly my current pipeline became much clearer: Astra hunts / plans. Search wide. Challenge assumptions. Find the ugly edge cases. Fable implements. Take the evidence-backed work, make the smallest coherent change, and get it shipped. Then Astra reviews and assumes “done” is probably hiding another layer, while Fable ships: consolidate, remove unnecessary intervention, and run the final implementation pass. But there’s still a problem: Astra can generate a lot of findings. Sending every finding straight into Fable is expensive, and some findings may be hardening, duplicates, or assumptions rather than actual contract breaks. So now I think Grok belongs between Astra and Fable, not as another coder, but as the triage/orchestration layer. For me now the steps are clear: 1-Astra produces findings + tests. 2-Grok classifies, deduplicates, and prioritizes them. 3-Fable receives only the evidence-backed implementation work. 4-Astra reviews the result. 5-Grok maintains state and routes the next step. And this is where Grok Bot gets interesting for me. I can give the bot tools for things like severity classification, contract checking, test evidence, duplicate detection, routing, and human escalation. The human still owns the final gate: Ambiguous requirement? Ask me. Destructive change? Ask me. Test contradicts product intent? Ask me. Otherwise, keep the pipeline moving. So after all of this, my “Round 3” conclusion isn’t another winner. It’s: Astra hunts. Grok triages. Fable implements. Astra reviews. Fable ships. Human approves. Maybe the mistake was benchmarking them as competitors in the first place, and the interesting system is the team. And through this whole process I’m also fine-tuning the cost side. I need to make sure Astra’s review and planning output is clean, requirements are clear, findings are well distributed, and Fable only gets the work that actually deserves implementation. So yes, after all these rounds, I basically came back to the same idea I started with: orchestration + customized routing for the actual project matters more than picking one “best” model. PS: Don’t get me wrong I didn’t put Grok in the loop just because I already pay for Cursor or because I wanted to force Grok Bot into the architecture. Grok actually fits a different role here. Astra can generate a lot of findings, but not every finding deserves expensive implementation. Grok can sit in the middle as the cheaper orchestration layer: classify severity, deduplicate issues, check evidence, maintain state, decide what should actually reach Fable, and escalate ambiguous cases to me and that’s where Grok Bot matters for me: not as “another coding model,” but as a persistent coordinator with tools and state, keeping the loop moving while the more expensive models are only called when their specific strengths are needed.