jevVIDEO

Jev vs OpenAI Decisions on spotting problematic agent steps

Across 30,943 steps in 5 real agent runs, Jev caught 253 problematic steps OpenAI missed (OpenAI caught 51 Jev missed) at 3.8x lower cost, though OpenAI was faster and more precise.

Calvin Huang

@calvinhhuanggArchived 22 days agoOriginally posted 2026.10.10

Calvin Huang @calvinhhuanggOriginally posted 2026.10.10

Open original on X

Jev vs @OpenAI's Decisions API: which is better at identifying the problematic steps in an agent run? We ran both on 30,943 steps in 5 real runs. Jev caught 253 problematic steps OpenAI missed (OpenAI caught 51 Jev missed) at 3.8x lower cost. OpenAI was faster and more precise.

x.com/calvinhhuangg/status/2108651470518448615Archived 2026.10.10 10:07