Jev vs TabPFN: a job fraud moderation test
On 17,880 job postings, Jev scored AUC 0.8883 with 44 examples while TabPFN reached 0.9995 on the full set, leading the author to suggest Jev makes decisions and a tabular model makes predictions.
We tested what happens if an agent moderates a job board and bans any post scored above 60% on fraud risk. This is a classic prediction (and decision!) task - few fraud examples, a lot of legitimate ones, and outcomes that cost you both time and money. We used an open dataset of 17,880 job postings and evaluated ROC AUC of two models - Jev and TabPFN - on the same 300 test posts: Jev, 44 examples (enforced by its context limit): 0.8883 TabPFN, 400 examples: 0.9485 TabPFN, all 17,567 eligible training examples: 0.9995 Jev’s average predicted fraud risk was 27.5%, while the actual rate was 5%. TabPFN landed at 4.8% with 400 examples. This suggests a new approach to agentic systems - the agent handles end-to-end moderation, Jev helps it decide on the right tools, while a tabular foundation model makes the predictions. And in case you'd like to try and compare yourself, the setup and results are in the cookbook - link in comments.