jevIMAGE
Reproduction finds Jev-style decision models are not probabilistically coherent
The author reproduced a paper on a weather forecasting task and found that flipping a predicate gives P(A)+P(not A)>1, and that decision models are sometimes worse than standard LLMs.
Shuvom Sadhuka
@shuvom_sArchived 19 days agoOriginally posted 2026.10.11
Shuvom Sadhuka @shuvom_sOriginally posted 2026.10.11
Decision models like Jev are getting a lot of hype right now, but are decision models I actually probabilistically coherent? What happens if you flip the predicate from "it will rain" to "it will NOT rain"? Turns out, not only do they give incoherent answers (e.g., P(A) + P(not A) > 1), but they're sometimes WORSE than standard LLMs. I recreated this work from @SuproteemSarkar and Isaiah Andrews on a simple weather forecasting task using OpenAI's new decision model.
This post quotes another post: x.com/suproteemsarkar/status/2095904452225990966
x.com/shuvom_s/status/2109013068609167479Archived 2026.10.12 10:05