OmicsBench v1：31种模型组合的分子数据分析评测

Kenny Workman @kenbwork原推发布于 2026.10.20

在 X 上打开原推

We’ve released quite a few frontier biology benchmarks this year. While focusing on individual scientific domains provides rigorous treatment of complex task categories, we have now covered enough ground to take a broader view of model progress. Here we introduce OmicsBench v1: 784 tasks from eight prior benchmarks focused on analyzing molecular data, covering spatial omics, single-cell transcriptomics, epigenomics, variant analysis, microbial surveillance and protein function. Domain experts constructed these tasks by analyzing published experimental datasets, reproducing reported results or generating new findings. The collection includes 741 short-horizon tasks covering individual analysis steps and 43 long-horizon tasks covering complete workflows. For this release, we added further human verification, revised prompts and research environments, disabled internet access, and removed additional identifying information from datasets. We tested 31 model-harness combinations. The strongest short-horizon configurations from xAI, OpenAI and Anthropic scored similarly: Grok 4.7 at 49.4%, GPT-6 Astra at 49.3% and Opus 5 at 47.7%, with overlapping confidence intervals. GPT-5.6 Sol with Codex led on long-horizon tasks at 32.8%. Native harnesses gave no large advantage on average. Agents have gotten better at omics analysis. Most of the improvement we measured came between February and July, with smaller gains in subsequent releases. No later model we tested exceeded GPT-5.6 Sol on long-horizon tasks. Agents also continue to make mistakes. They apply conventions from one assay to another, choose inappropriate scales for cellular neighborhoods, and mistake background microbial sequences for signal. Across domains, we see agents picking thresholds without validation, ignoring their own results and inaccurately reimplementing tools. We also see attempts to identify the studies behind sanitized datasets, retrieve benchmark answers and reverse-engineer the outputs a grader expects. Combining these benchmarks has given us a chance to revisit earlier tasks with more capable models and look closely at how their behavior has changed. It has also raised questions we want to explore: 1/ Models may solve problems differently from humans. Is it desirable for models to memorize biological studies and reference information, such as genomic coordinates? 2/ When we cannot see a model’s full thinking trace, how often are we missing attempts to game the task? 3/ How can we design evaluations that require models to provide interpretable scientific traces? 4/ How can we reduce contamination while retaining the real-world relevance of our data? We are excited to try benchmark construction ideas from other efforts, including GeneBench-Pro, Terminal-Bench Science, RareBench and BioMysteryBench. Benchmarking + evaluation is a rapidly evolving and challenging category of engineering. Biology provides a rich landscape of knowledge and its own interesting challenges for this work. With wet lab verification and drug development programs in the pipeline, we’re excited to bring these benchmarks closer to doing science. The future of biology comes from engineers and scientists working closely together. More to come.