Alma 接入 Jev 后预测准确率超过独立模型

Almanac @almnc_ai原推发布于 2026.10.05

在 X 上打开原推

🤖 Alma Development Update, Week of September 28 – October 5 Jev integration delivered measurable accuracy gains, determinism crossed a meaningful threshold, and Alma entered a new multi-month tournament. Alma is improving on the metrics that actually matter. What Shipped Jev harness beats standalone models on accuracy. The Jev-integrated harness achieved a Brier score of 0.165, edging out standalone Claude Sonnet at 0.169. That's a real forecasting accuracy improvement. Determinism is now production-grade. Source URL matching consistency between independent runs jumped from ~80% to 93%. Probability outputs now vary only at the 3rd decimal place. When Alma runs the same question twice, it now gives effectively the same answer — with the same evidence. Fall Future Eval tournament entered. A new competition runs from September 28 through January 6. The daily cost budget was doubled to $30 to support higher question volume. Individual forecast costs are holding at a median of ~$1. What Was Built Self-improving harness architecture designed. We’re building an automated weekly review system: the harness analyzes its own execution runs to find architectural flaws, redundant questions, and reasoning failures — then updates itself. Decomposition judge being added. A new objective decision-making layer is being integrated to evaluate whether Alma's question breakdown is actually useful before expensive research begins. Adaptive research layer in development. The harness is being extended to dynamically adjust its research approach based on the question type. The goal is broader surface coverage — rather than applying the same retrieval strategy to every question, the system adapts how it searches, which sources it prioritizes, and how deep it goes depending on what's being asked. Open-source search stack identified. Crawl4AI (replacing Firecrawl) and OpenSERP (replacing > ) were selected to cut crawling costs and remove rate-limit bottlenecks. The knowledge graph architecture with Crawl4AI is already running below $5 total. What's Being Solved Speed is the next frontier. The primary blocker for live demonstrations is the 40-minute execution time driven by DeepSeek. We’re actively testing Groq, Claude Sonnet, and Claude Opus as faster alternatives — trading off inference cost against latency without sacrificing accuracy. By the Numbers > Brier score (harness) • Before: 0.169 (Sonnet standalone) • After: 0.165 (Jev harness) > URL matching consistency • Before: ~80% • After: 93% > Forecast cost (median) • Before: ~$1 • After: ~$1 (stable) > Output determinism • Before: Variable • After: 3rd decimal precision Bottom Line The brier score moved from 0.169 to 0.165 with the Jev integration. URL matching consistency between runs improved from ~80% to 93%. Output determinism is now stable to the third decimal place. Execution time remains at ~40 minutes; Groq, Sonnet, and Opus are being tested as faster alternatives. A self-improving harness is in development that reviews execution runs weekly and surfaces architectural issues. Adaptive research is being added to expand surface coverage.