用 Jev 替代 embedding 分类器做数据清洗

Bo @bo_wangbo原推发布于 2026.10.08

在 X 上打开原推

A lot of the teams rely on embedding model to build high-quality pre-training data. For example: Fineweb-edu is built from Fineweb, one key cleaning process is the fineweb-edu-classifier. A 6-class rubrics to quantity text quality. I sampled 1k Fineweb-2 pages, compared Jev/pplx-decider-v1.1 and Fineweb-edu-classifier against Astra annotation. Conclusion: decider is also a quality data judger, works out-of-the-box.