Using Jev as a judgment layer in RAG and evaluation
The author explains how Jev fits into a RAG pipeline: one call scores 20 chunks with individual probabilities, keeps those above a threshold, then checks whether the evidence is enough, and describes jev-as-judge grading support answers on four questions.
I looked at how people plug Jev into a RAG pipeline, and the idea is simpler than it sounds. Most bad RAG answers are a judgment problem. Hybrid search finds a shortlist. Twenty chunks that look related to the question. It can't tell which of them actually contain the answer. So the LLM gets all twenty and guesses. That's where made-up answers come from. Jev sits in the middle. It asks one yes or no question about every chunk, all in one call, and returns a probability for each. Your code keeps what's above the threshold and drops the rest. Then one more check, whether the kept chunks are enough to answer at all. If they're not, the app says "not in the documents" instead of making something up. A few things still matter: → Search sets the ceiling. If the right chunk isn't in the top twenty, nothing after it can bring it back. → The threshold is a product decision, not a model setting. → An injection score is one more filter, not security. If you want to see the same idea in real code, look at jev-as-judge. It's not a RAG demo. It uses Jev to grade support bot answers on four questions: → is every claim backed by evidence → does it answer what was asked → did the bot really do what it says it did → is it helpful The results go into Comet Opik as an experiment. Three things in it I'd copy into any eval setup: → cheap code checks run before the model is even called → a "needs review" zone between pass and fail, so nothing gets forced into a yes or no → the raw model response is saved with the rubric version, so any score can be checked later Let a small model judge, let your code decide, and treat "I'm not sure" as a real answer. That's what makes the whole thing trustworthy.