Author rebuts claims that Jev reranking benchmark is flawed
A practitioner argues that a Jev reranking comparison based on 80 SciFact queries and ndcg@10 is inconclusive, listing seven reasons, while still calling Jev likely the SOTA pointwise reranker.
Unfortunately almost everything about this tweet is a mix of incorrect or inconclusive >80 SciFact queries >ndcg@10 1. 80 queries is nowhere near enough to draw a statistically significant conclusion, esp. about reranking where performance differences between models aren't typically that large 2. ndcg@10 is an irrelevant metric in 2026 3. latency likely varies significantly by location or current Jev workload, I've also seen latency stats where Jev outperformed other rerankers like cohere 4 fast and voyage rerank 2.5 on p50 and p95 latency 4. SciFact queries are short factual statements, dissimilar to common query shapes you'd see from a search agent 5. SciFact's corpus has 5,183 documents in it, so there aren't going to be as many hard negatives for a given query to test reranking capabilities on 6. It's probably worth running the latest SOTA llms over scientific search tasks like this to see where they disagree with the existing ground truth on an IR dataset from 2020 with human labels to determine whether or not the ground truth needs fixing 7. Baseline problems - customers in the US and EU probably aren't comparing to a closed source qwen reranker you can only access via Alibaba's cloud offering at the moment. Most of them are likely unaware this new qwen reranker exists and it's unlikely its competitive with voyage rerank 3 for example Please be careful assuming the benchmarks/evals you see online like this apply to your workload. I'm still convinced it's more likely than not that Jev is an SOTA pointwise reranker when you measure it more accurately
This post quotes another post: “Milvus tests Jev as a RAG reranker replacement”