Jev in practice: make the small decision first and split one question into five
Putting Jev in front of an expensive model cut a 120-ticket test from 22 minutes to 42 seconds, and splitting a phishing question into five raised accuracy from 89.4% to 95.0%.
4 things to do when you get access to Jev: 1. don't replace your model, put Jev in front of it Jev makes the small choice first, and your expensive model only runs when it's really needed one email tool swapped 2 AI calls per email for 1 Jev call, and kept all their normal safety checks on a 120 ticket test: 42 seconds with Jev in front → 22 minutes without it cost: $0.0003 → $0.059 the person who ran that test said his slow version was slowed down even more by retry errors, so the real gap is smaller how to do it: - look through your code for AI calls that only choose something and never write text - write the list of possible answers in your own code, don't let the model invent them - send that list to Jev, and keep your old call as a backup - put it behind an on/off switch so you can undo it in one line - run both for a week and compare them before you delete the old one you're not rebuilding your product, you're replacing one call 2. ask 5 small questions instead of 1 big one someone tested Jev on 2,000 phishing emails ask it one big question and it gets 89.4% a simple 2 line text rule gets 91.8% Claude Haiku 4.5 asked the same question gets 94.2% so when it has to give the final answer alone, Jev loses even to a text rule then he asked 5 small questions instead, and added the answers up in his own code 95.0%, the best score in the whole test SAME MODEL, SAME EMAILS how to do it: - take the big question you were about to ask - write down the 5 things a person checks before they answer it - ask each one separately, all in the same request, they run at the same time and cost almost nothing extra - add the answers up in your own code, with your own weights - test those weights on 100 examples where you already know the right answer - change the order of your options and run it again, reordering 4 options changed 7 answers out of 120 Jev is good at noticing things and bad at making the final call so keep the final call in your code 3. never ask Jev if it can answer on 120 test tickets, any question like "do you have enough info?" said yes on 85% of them a simple "need more info" flag said yes on 119 out of 120 tickets, and it then answered those tickets about 87% correctly it doesn't know what it doesn't know how to do it: - delete any question like "can you answer this" or "is there enough context" - make every option a real action your code can run - always read the confidence score, not just the answer - choose your limit by risk: low for reading data, 0.85+ for anything you cannot undo - send everything below the limit to a human or to your big model in that same test, using 0.8 as the limit passed 30 cases to a human, and 93% of them were passed for the right reason 4. run it quietly next to what you already have there's already a langchain package published and a pydantic-ai adapter being reviewed, so this is closer to a settings change than a rebuild how to do it: - leave your current system in charge, it still makes every decision - send the same input to Jev too, and throw its answer away - save both answers plus Jev's confidence score into one table - after a week, look only at the rows where the two disagreed - switch over only for the cases where Jev was right that list of disagreements becomes your test set, and your normal traffic builds it for free and don't use it when you already have labelled data if the question never changes and you have examples to train on, a small model you host yourself beats Jev on speed and price, and needs no API key at all Jev wins when you have no labelled data and the question keeps changing so use it where the list of answers is short and the question is boring that's most of your agent anyway everything above comes from other people's public tests, not from production, because the model is only 4 days old i'm just sharing what i find while i test this and try to make the work in my own company faster and cheaper tomorrow i'll show you what happened when i put Jev in front of the meta ads work we do for one big client and most of you are still on the waitlist anyway so start with step one, because it needs no access and no API key at all: find the calls in your code that were never writing tasks in the first place gl https://x.com/CompleteSkeptic/status/2099925684256899543/video/1
This post quotes another post: “Jev setup guide: replacing unnecessary LLM calls with Choice and Score”