Jev confidence vs. probability: what should your code act on?

Use the returned answer to understand the model’s judgment, then apply a separately evaluated policy. A confidence number is not permission to perform an action, and it is not an independent certificate of correctness.

TypeSafe’s confidence documentation recommends making uncertain cases take a different path. The specific boundary depends on the task and the consequences of a mistake.

Read the right field

Choice returns a selected label, a distribution across labels, and a separate confidence value. Score returns an expected position along its rubric, level probabilities, and confidence. Noul returns the probability of yes without a separate confidence field.

These fields are related, but their names are not interchangeable. In particular, a Noul value near zero means a likely no answer; it does not mean the model has no confidence.

For an urgency question, a low probability of urgency might support an ordinary queue. Whether that is acceptable depends on how costly a missed urgent case would be.

Distinguish a signal from an action

A router can predict billing but still send the message for review. That can happen because its confidence is below your threshold, because the input is incomplete, or because the requested action requires human confirmation.

Keep these conditions visible in code. They are product rules. Changing them should not require hiding another instruction inside the model question.

The uncertain-classification example shows this separation. Its numeric cutoff demonstrates the mechanism; it is not a recommendation for every application.

Choose thresholds with labeled examples

Start with cases that resemble the traffic you expect, including ambiguous and out-of-scope inputs. Keep development examples separate from the cases you use for your final evaluation.

For each candidate threshold, inspect:

• How many cases would proceed automatically.
• How many automatic decisions are wrong.
• Which kinds of wrong decisions remain.
• How much work moves to the fallback path.

A stricter cutoff may reduce coverage without eliminating the most expensive errors. Review individual failures before deciding that the tradeoff is acceptable.

Do not transfer cutoffs blindly

Changing a model version, question wording, label set, or rubric changes the evaluated system. Recheck the policy after such changes.

TypeSafe’s Jev 1.13 notes also caution against assuming arithmetic identities between separately phrased questions. A question and its negation should not be treated as a guaranteed complementary pair. A threshold tuned for one primitive should not simply be transplanted to another.

Plan for more than uncertain answers

An API request can also fail, time out, or return an answer your application cannot use. Define a safe fallback for those operational cases independently of the classification cutoff.

For an early integration, logging a recommendation for review is often easier to evaluate than automatically executing it. The urgent-ticket example reports a probability while leaving the ticket in a review queue. That lets you collect evidence before deciding how much automation to introduce.