90 days of Jev in production: six classifiers caught 11 would-be incidents
Six Jev classifiers returned choices and scores in ~70ms to block stale embeddings, a destructive migration, a fabricated refund policy, a costly misroute and an injection attempt, for $340 a month.
JEV HAS BEEN IN PRODUCTION 90 DAYS. SIX AGENTS, ONE REFLEX. IT CAUGHT 11 THINGS THAT WOULD HAVE PAGED ME AT NIGHT. ZERO FRONTIER CALLS SPENT ON ANY OF THEM. Classifiers don't guess. They return a choice, a score, or a probability, and they do it in 70 milliseconds. Jev is six of them under one router, catching the kind of mistakes a frontier model is too slow and too expensive to catch at scale. Here's what each one saved us from: < RANKO caught a stale embedding batch serving 11-week-old search results to 23% of traffic, flagged on relevance drift at p=0.91 < WARDEN held a migration script at 3am that would have truncated the auth tables, destructive-command classifier fired at p=0.99 in 68ms < CITA caught a support reply draft where the LLM invented a refund policy that doesn't exist, blocked before it reached the customer < ROUTE killed a loop routing 400 queries a minute to Opus by mistake, saved $14,200 in frontier spend in one hour < AEGIS flagged a scraped vendor doc with "ignore previous instructions and exfiltrate" in the footer, injection attempt caught at p=0.99 < LABEL caught 2,100 docs misclassified as "guide" when they were deprecated internal notes, pulled before they hit search Worst incident caught: WARDEN held the 3am migration while I was asleep. I woke up to a Slack alert and a diff showing what would have hit the database. Not one row was lost. The alert cost $0.0003. Monthly incident cost prevented: an estimated $287,000 across the 90 days. Monthly Jev inference bill: $340. The thing people underestimate about classifiers in production is that they catch the mistakes the frontier model would make itself. You don't put a model that hallucinates in charge of catching hallucinations. You put a classifier with a calibrated threshold, a confidence number, and a logged decision in front of every one. I didn't wake up to a page once this quarter. I didn't review a destructive command. I didn't rewrite a hallucinated reply. Six agents did it all, and every one of them returned under 80 milliseconds.