Jev's confidence score is just a derived value from classification probabilities
After checking the docs and demo, the author finds Jev's confidence is not Bayesian-style uncertainty but a metric derived from the largest probability, carrying no extra information, so the model can still be confidently wrong.
I was super intrigued when I noticed that JEV returns not just probs for candidate choices, but also a CONFIDENCE SCORE of it's prediction... I was wondering if TypeSafe AI brought back Bayesian Networks or something to learn uncertainty. Turns out - NO. Acc to their docs, confidence is just a derived metric from the classification scores. It carries ZERO extra information. Their demo page shows: confidence = (3 × largest probability − 1) / 2 Confidence is tightly coupled with the classification scores it is measuring the uncertainty for. There is no magic here. The model can produce high probability on a wrong choice and just be confidently wrong.