Jev replaces a multi-model ensemble for code review: zero false positives, 50x faster
The author swapped an internal automated code review from a GLM, Grok and Gemini ensemble to Jev scoring git diffs first and calling an LLM only on hits, getting zero false positives, seconds instead of minutes, and about 100x lower cost.
Ok, at first I thought @typesafeai was a bit overhyped because essentially "there is nothing new" in doing json schema predictions. After seeing all the buzz around it, I wanted to take it for a spin as well so I took one of our internal @scriptdotit workflows for automated code reviews and wanted to benchmark Jev against it. And it's good...very good actually. We have been using our @scriptdotit workflow for some time now so we have gathered data around false positive and false negative reviews from the AI reviewer. We also used many different LLMs over time. We believed in diversity of reasoning so we tried and tested many models over time: Gemini, Claude, GPT, Grok, GLM and more. An important point about code review AI reviewer workflows is that they need to be fast, cheap and correct. We don't want it to miss important comments (false negatives), we don't want it to give wrong reviews (false positives), we don't want to wait a long time for the review after each commit (latency) and since it's running very frequently (kudos to the team :) ) then it needs to be cheap. Now the results, I benchmarked it against our current setup which is an ensemble (GLM + Grok + Gemini) multi-turn code reviewer. To use Jev properly, I had to change the workflow architecture a bit. Instead of letting an agent "explore" the PR and propose issues, we score the raw git diff directly with Jev first, and only invoke an LLM to format an explanation when Jev flags an issue. The benchmark on our data showed: - Zero false positives: It completely filtered out the false positives that the developers pushed back on - ~50x faster: Reviews dropped from minutes down to just a few seconds. - ~100x cheaper: A massive cost reduction compared to running multi-model agent ensembles. - 75% overall bug recall: It caught three out of every four confirmed issues without needing multi-turn exploration. As you can see, it's a huge leap, but there are trade-offs. The agent ensemble workflow picked up more bugs overall, but the speed and cost improvements here are impossible to ignore. Overall, it turns out you don't always need heavy, multi-turn model agents to achieve high-precision code reviews. Using calibrated probability scoring with a good model like Jev gets you the vast majority of the value at a fraction of the time and cost. I'll write about it more in the future and if you would like to test the actual workflow we are using, let me know and i'll share it with you.