WHAT NEW IN X11.05 THU

X 上的 AI 合集档案 · 23 个合集 · 69,408 条

jevIMAGE

Momentic 实测 Jev 定位元素:51.8% 准确率、0.35 秒

Momentic CTO 在 Locator Bench 上测 Jev:单用 51.8% 准确率、0.35 秒;加 v5 兜底到 78%,仍低于自家 v5-auto 的 84.6%。

wei-wei
@wuweiweiwu2026.09.25收录原推发布于 2026.09.25
Momentic 实测 Jev 定位元素:51.8% 准确率、0.35 秒
wei-wei @wuweiweiwu原推发布于 2026.09.25
在 X 上打开原推

our CTO @itsjeffan tested @TypeSafeAI's Jev on Momentic's hardest locator eval Locator Bench the set requires visual, spatial and logical reasoning to find the element a user describes Jev got it right 51.8% of the time at 0.35s median latency. fast, but below the accuracy our customers need on the same benchmark: - Jev + v5 fallback: 78.0% accuracy at 1.5s median latency - Momentic v5-auto: 84.6% at 2.5s on our separate locator + assertion evals, Jev was 8x faster than Luna, our fast config, at 1/4 the cost accuracy was lower there too the Jev-only prototype was more limited on our broader AI-action eval. it could click and type, but couldn't run JavaScript, do AI checks or extract data. it ran 4x faster per step than Luna, but only completed ~8% of the tasks with Luna handling general tasks and Jev locating elements, completion reached ~24%. that hybrid ran 2.8x faster per step than baseline we've already broken general tasks into smaller classification problems for our sub-agents, so i can see Jev being useful there but it can't take images, and even checking whether a page is ready can need visual context it's impressive for a first gen. we'd still need it to meet our accuracy requirements for each task before using it in customer tests if a test keeps failing for the wrong reasons, engineers learn to ignore it that's when it's worse than having no test

x.com/wuweiweiwu/status/2103188520156229672采集于 2026.09.25 10:04