Developer says Jev's returned confidence scores are misleading
A critique argues Jev's API reuses the network's output probabilities as confidence, which carries no Bayesian data uncertainty and can lead developers to overestimate it.
Love Jev but what the API returns as confidence is extremely misleading - and I feel many devs are unknowingly overestimating what it actually is. They reuse the network's output probs to calculate these so-called "confidence scores". Meaning the same weights and compute graph generate both the probabilities and the confidence. Therefore, this confidence score does NOT measure data uncertainty in a bayesian sense. It carries no extra information. Network can be confidently wrong coz the same process produces both the result and the confidence. To their credit, typesafe ai docs do acknowledge this.
This post quotes another post: x.com/typesafeai/status/2106197282274324846