jevcal

What it does

jevcal runs typed decision questions against labeled examples, measures how confidence relates to correctness, and selects thresholds intended to meet a target on held-out data. It writes a decision lock file and an HTML report, and can remeasure the locked configuration in CI to flag accuracy, coverage, or model-version drift.

What you can reuse

The portable artifact is the bundle around a question: questions.yaml, labeled JSONL, saved predictions, thresholds, and evidence. The split between measurement and compilation means previously collected predictions can be inspected without spending on another model call. A cascade runtime can pass low-confidence items to a fallback and fail closed when none exists.

How it fits

Representative states and labels become an evaluation set. Jev produces typed answers and probabilities. Deterministic evaluation code selects thresholds, reports coverage and errors, and decides when a slower fallback is needed.

Setup and compatibility

The documented workflow uses Python 3.10 or newer and installs from GitHub. A no-key demo uses a deliberately imperfect simulator so a user can inspect the report format before connecting Jev.

Limitations

The sample metrics in the README are simulator output, not a Jev benchmark. The author recommends roughly one hundred or more labeled rows per question and warns that labels, ambiguous questions, model changes, and rounded probabilities can all affect a threshold. Jev Directory has not reproduced the author-reported live verification.

Sources

Primary source: jevcal repository.