Expand description
v0.34: Calibration scoring over resolved predictions.
A Prediction carries the predictor’s confidence in the expected
outcome (a number on [0, 1]). When a Resolution records what
actually happened, the resolver also records matched_expected
(a bool). Together those two facts let us compute, per actor:
- Hit rate: fraction of resolved predictions that matched.
- Brier score: mean of
(confidence - matched)^2across the resolved subset, wherematched ∈ {0, 1}. Lower is better. Brier = 0 means perfect calibration; 0.25 is a chance-level binary predictor; 1.0 is maximally wrong. - Log score: mean of
log(p_assigned_to_actual_outcome). We clip to[1e-9, 1 - 1e-9]to avoid-∞. Higher (closer to 0) is better.
These are derived signals — never written to disk, always
recomputed from the canonical predictions and resolutions
collections. That keeps the kernel ledger source-of-truth and
avoids stale calibration cache concerns.
Calibration is the move that makes Vela an epistemic ledger rather than a knowledge graph: every actor accumulates a public, reproducible track record of how well their stated beliefs match reality.
Structs§
- Calibration
Record - Per-actor calibration summary computed over the resolved subset of the actor’s predictions.
- Expiration
Report - v0.40.1: report from one expiration pass.
Functions§
- calibration_
for_ actor - Convenience: calibration for a single actor.
- calibration_
records - Compute calibration records for every actor that has at least one prediction in the frontier.
- expire_
overdue_ predictions - v0.40.1: walk every prediction in the project and mark as
expired_unresolvedany whoseresolves_byis in the past and has no associatedResolution. Emits oneprediction.expired_unresolvedevent per newly-expired prediction.