Jev 发票分类实测：50 份文档对比六个模型

Paweł Huryn @PawelHuryn原推发布于 2026.09.19

在 X 上打开原推

One developer’s list of projects built on Jev grew from 46 to 160 in three days. I wanted a measurement to go with the excitement. Jev takes text, a question, and a list of possible answers. It picks one and reports its confidence. I tested the vendor’s showcase task: sorting invoices. My first dataset was too easy - all six models scored 100%. I rebuilt it. Fifty documents across six types. In 32, the obvious clue is a lie. A “PROFORMA INVOICE” billing for goods already shipped. A purchase order with prices that make it look like a quote. An “INVOICE SUMMARY” explicitly saying it isn’t a request for payment. Ten languages. OCR errors. An amount written out in words. Three documents without obvious keywords, including a freelancer’s invoice written as an email. Correct answers, followed by cost per 1,000 decisions: Jev: 50/50, $0.025 GPT-OSS 20B: 48/50, $0.030 Ministral 8B: 48/50, $0.031 Claude Haiku 4.5: 50/50, $0.39 Gemini 3.8 Flash: 49/50, $1.03 Claude Opus 5: 49/50, $2.83 The “444× cheaper” claim compares Jev with Fable 5.1 and GPT-6 Astra. My Opus comparison also produced a huge multiple, but Opus isn’t the most useful baseline here. Against Ministral 8B, which you can run locally, Jev cost about 20% less and took about 90ms longer per decision, sending one question at a time. Batched, Jev answered 32 questions in roughly the time it answered one. Then I removed the definitions and gave Jev only the six category names. It got 46/49 completed requests right; one API call failed. Every mistake fell below 0.80 confidence. Correct answers averaged 0.97. A 0.80 cutoff would have flagged all three mistakes. Nothing wrong got past the line in this run. The failed call needed separate handling. Then I tested 24 support messages with house rules that contradicted the obvious answer - duplicate charges going to support instead of billing, for example. Rules in the prompt: 24/24 correct. Rules left unstated: 5/24. Of the 19 wrong answers, 15 came back above 0.90 confidence. Certain, and wrong. Confidence caught every invoice error here. It did not reliably flag errors caused by missing business rules. Those rules must be supplied with each request. Corrections don’t carry over between calls on their own. I also tried labeled examples instead of rules: Jev scored 13/24; Ministral 8B scored 18/24. For real invoice sorting, you want three things: A record of each input, decision, and confidence score. A threshold for sending uncertain decisions to a person. A system that improves as humans correct it. Jev offers a starting point for the first two. The third needs a feedback loop: collect corrections, update the system, and test the changes. A dedicated classifier with a retraining pipeline is one established option. Before you have training data, when your rules are in plain English and you need something running this afternoon, Jev looks useful. In my invoice test, it matched the best accuracy at the lowest cost. You still have to write down your rules. High confidence won’t tell you which ones you forgot.