Jev tested on 3,548 pages of real documents: retrieval helps, verification falls short
Testing Jev on real pipeline data, the author found reranking raised BM25 hits from 11/24 to 18/24, but conclusion checks produced 3 false warnings out of 32 cases and missed a greater-than-or-equal error, so it is not being adopted yet.
I tested Jev on real outputs from our pipeline and their source documents (3,548 pages) to see if the hype translates into something useful for us. For now I'm not adding it. The most promising part was retrieval. On 24 searches, BM25 alone put a relevant page first 11 times. With Jev reranking the same results it went up to 18, which is a real improvement. Using it to check conclusions was less convincing. I gave it 32 hand picked examples where I already knew the answer. It raised 10 warnings and 3 of them were on answers that were correct. It also let through a "greater than or equal to" where the source says "greater than". And 3 of those 4 mistakes came with confidence above 0.8, so I can't just filter on confidence. The problem with a warning is that something has to check it before acting on it. If I trust it blindly I end up changing things that were already right. The whole test cost about two cents and answers come back in one second, so I understand why people are excited. But a cheap API doesn't help me much if I have to double check what it tells me. It's a small test, but I think the hype is ahead of the evidence. For our use case it hasn't earned its place yet.
This post quotes another post: “ChatGPT co-inventor releases Jev, a model built for decisions rather than text”