Replacing embedding classifiers with Jev for data cleaning

Bo @bo_wangboOriginally posted 2026.10.08

Open original on X

A lot of the teams rely on embedding model to build high-quality pre-training data. For example: Fineweb-edu is built from Fineweb, one key cleaning process is the fineweb-edu-classifier. A 6-class rubrics to quantity text quality. I sampled 1k Fineweb-2 pages, compared Jev/pplx-decider-v1.1 and Fineweb-edu-classifier against Astra annotation. Conclusion: decider is also a quality data judger, works out-of-the-box.