Open basal-omni-ultra beats Jev 1.14 on Werdykt v1 benchmark
The author of the open basal decision models reports that basal-omni-ultra scores 0.879 macro accuracy on his 5,000-item Werdykt v1 test versus 0.822 for TypeSafe's Jev 1.14, leading in 9 of 10 areas, with the leaderboard and samples public pending independent retesting.
Remek Kinas (@KinasRemek), author of the open basal decision models, showed that his basal-omni-ultra scores 0.879 macro accuracy on the Werdykt v1 test against 0.822 for TypeSafe’s Jev 1.14, and leads in 9 of 10 areas. The model handles text, image and sound in Polish and English, plus an extended query schema Jev does not have (multi, act, tool), and it scores 0.928 on agent and tool decisions. Someone will ask whether this is the author’s own test, and yes, but it consists of 5,000 hidden items, the same ones go to every model, each gets one call per item, and the leaderboard and samples are public at http://basal.si5.pl. The release is on Saturday, so I am waiting for independent measurements. In the one area where it does not lead, real documents, everyone sits around 0.99, so it is a tie up to the rubber stamp.
This post quotes another post: x.com/kinasremek/status/2110651784033456229