3548 页真实文档实测 Jev：检索有用，校验不够可靠
作者用自家流水线真实数据测 Jev，重排把 BM25 命中从 11/24 提到 18/24，但结论校验 32 例中误报 3 例、漏掉一处大于等于，暂不接入。
I tested Jev on real outputs from our pipeline and their source documents (3,548 pages) to see if the hype translates into something useful for us. For now I'm not adding it. The most promising part was retrieval. On 24 searches, BM25 alone put a relevant page first 11 times. With Jev reranking the same results it went up to 18, which is a real improvement. Using it to check conclusions was less convincing. I gave it 32 hand picked examples where I already knew the answer. It raised 10 warnings and 3 of them were on answers that were correct. It also let through a "greater than or equal to" where the source says "greater than". And 3 of those 4 mistakes came with confidence above 0.8, so I can't just filter on confidence. The problem with a warning is that something has to check it before acting on it. If I trust it blindly I end up changing things that were already right. The whole test cost about two cents and answers come back in one second, so I understand why people are excited. But a cheap API doesn't help me much if I have to double check what it tells me. It's a small test, but I think the hype is ahead of the evidence. For our use case it hasn't earned its place yet.
这条是引用着另一条写的：「ChatGPT 联合发明人发布新模型 Jev」