Three days of Jev in production, failure cases included
The author describes actually using Jev on production systems for three days through mapping, shadow runs, measurement, and promotion, and lists the failure cases along with plans for further evals.
What this article says
这不是演示,而是作者在生产系统上连续使用 Jev 三天的复盘,明确把失败案例也写进去。他总结出一套四步工作流:map(把现有决策点映射成 Jev 可表达的结构化选择)、shadow(让 Jev 与线上原逻辑并行跑、只观察不接管)、measure(用评测指标对比两者结果)、promote(指标达标后才把决策权交给 Jev)。文章强调这套流程的意义在于把模型替换从一次性切换变成可回滚的渐进升级,并给出各阶段踩到的坑。后续计划接入 Vercel agent-browser、Pydantic AI 和 trycua,继续补充评测数据。
How I'm actually using Jev from @typesafeai. No demo: three days on production systems, misses included.
Map, shadow, measure, promote.
With @vercel agent-browser, @pydantic AI, and @trycua next. More evals coming.
More: https://yonyon.ai https://x.com/i/article/2101233808896831489