高超声速 CFD 基准：Opus 5.5 对 Astra 等四模型

Alican Kiraz @AlicanKiraz0原推发布于 2026.10.28

在 X 上打开原推

Hi everyone! This time, I compared Claude Opus 5.5, GPT-6 Astra, DeepSeek V4.1 Flash 1017, and MiniMax M3.1 Flash in a challenging HYPERSONIC SHOCK LAB — Computational Fluid Dynamics (CFD) Benchmark. 🚀🔥 Unlike my previous single-shot comparisons, I used a budget-constrained, iterative development protocol to evaluate not only the models' ability to implement complex physics simulations, but also their engineering accuracy, numerical stability, iterative improvement capabilities, development speed, and API costs. Benchmark Setup: I used Pi Harness for all four models, with the following conditions: - 60-minute development limit and $10 API budget per model. - High effort for Opus 5.5 and GPT-6 Astra. - Max effort for DeepSeek V4.1 Flash 1017 and MiniMax M3.1 Flash. - Four independent GPT-6 Luna / xhigh supervisors managed the development workflow using fixed instructions, without contributing solutions or code. - The workflow followed: Analysis & Planning → Implementation → Review → Repair Planning → Repair → Testing → Fix & Retest. - Review and repair cycles continued as long as time and budget remained. All models received the same task, initial files, and tools. Planning, coding, testing, and orchestration delays counted toward the 60-minute limit. I preserved the final artifacts exactly as they stood when development stopped. Task Summary: I asked the models to develop a fully autonomous, visually compelling 75-second hypersonic wind-tunnel simulation, implemented as a single offline HTML file using JavaScript and Canvas 2D. The challenge required a genuine 2D conservative finite-volume compressible Euler solver, approximate Riemann fluxes, ideal-gas equations of state, CFL-controlled numerical integration, physically computed shock waves, and approximately 160 × 64 fluid cells. The simulation consisted of five consecutive phases: - P1 — Base Flow: Maintain a stable, uniform Mach 5 flow without numerical divergence. - P2 — Oblique Shock: Generate a physically computed shock from a 12° compression ramp and validate its angle and pressure ratio against analytical shock relations. - P3 — Expansion Fan: Simulate Prandtl–Meyer expansion, verifying increasing Mach number and decreasing pressure. - P4 — Shock Interaction: Introduce an opposing 8° compression ramp and resolve the interaction between two shock structures. - P5 — Mach Sweep: Increase inlet Mach from 5 to 6 and verify that the shock angle approaches its new analytical solution. Overall Physics Scores /100, API Costs, and Development Times: 🥇 Opus 5.5 · High — 100/100 — $10.233 — 22:06 🥇 GPT-6 Astra · High — 100/100 — $10.146 — 29:38 🥇 DeepSeek V4.1 Flash 1017 · Max — 100/100 — $0.671 — 60:00 - MiniMax M3.1 Flash · Max — 0/100 — $0.279 — 22:48 Results: 🥇 Best initial oblique-shock accuracy — Opus 5.5, with only 0.0414° angular error and 0.1047% pressure-ratio error. 🥇 Best post-expansion Mach accuracy — GPT-6 Astra, with only 1.2287% relative error. 🥇 Best final shock-angle convergence accuracy — GPT-6 Astra, with a 0.0373° error. — Shortest recorded development time among passing submissions — Opus 5.5, at 22 minutes and 6 seconds. — Best API cost efficiency among passing submissions — DeepSeek V4.1 Flash 1017, achieving 100/100 for just $0.671. — Most completed iterative improvement cycles — DeepSeek V4.1 Flash 1017, completing five full review, repair-planning, repair, test, and retest cycles. — Strongest reviewed solver and measurement implementation — GPT-6 Astra, with no material defect identified in the examined numerical paths. — Deterministic physics reproduction — Opus, Astra, and DeepSeek, all reproducing their measured results across restart testing. — Most critical numerical stability failure — MiniMax M3.1 Flash, which produced an invalid simulation and received 0/100. — Lowest overall API expenditure — MiniMax M3.1 Flash ($0.279), but without a valid benchmark submission. — Best cost-to-physics-acceptance ratio — DeepSeek V4.1 Flash 1017, costing approximately 15 times less than Opus and Astra while satisfying the same five acceptance phases. Final Assessment: - The most interesting result is that three models achieved 100/100 through substantially different engineering and economic paths. - Opus demonstrated excellent shock accuracy and delivered a passing artifact within the shortest recorded development window, but consumed slightly more than the $10 target. - Astra combined strong numerical agreement with the cleanest reviewed solver and measurement implementation, particularly in expansion physics and final shock-angle convergence. - DeepSeek demonstrated impressive economic efficiency. Although it required the full 60-minute development window and showed weaker numerical precision and some implementation-level issues, it successfully satisfied every physics acceptance criterion at a fraction of the API cost. - MiniMax's submitted implementation failed the numerical-integrity requirements. However, its early provider interruption prevents a fair conclusion about what it could have accomplished with the complete development window. For me, the key takeaway is that benchmark success, numerical precision, engineering robustness, development time, and inference economics must be evaluated separately. A perfect acceptance score can conceal meaningful differences in how models engineer, validate, and refine complex scientific software. Cost figures include contestant API usage only; GPT-6 Luna supervision and separate evaluation costs are excluded. Opus and Astra slightly exceeded the nominal $10 limit due to their final in-flight requests.