Expand description
Versioned, provider-independent offline decision evaluation.
Structs§
- Acceptance
Point - Accuracy and coverage at one observed confidence cutoff.
- Calibration
Result - Per-question calibration suggestion, never applied to production configuration.
- Comparison
Report - Read-only comparison result; regressions do not change the process exit code.
- Completeness
- Completeness counts; failures and missing records never count as incorrect predictions.
- Evaluation
Report - Deterministic offline evaluation report.
- Fixture
Case - A labeled request, with language, risk, and provenance for grouped analysis.
- Fixture
Suite - A versioned set of labeled requests. It contains no provider credentials.
- Metric
Regression - One changed metric in a comparable evaluation pair.
- Question
Metrics - Metrics for one question and optional language/risk slice.
- Saved
Responses - Versioned saved provider results for fully offline evaluation.
- Saved
Result - One saved case result.
Enums§
- Calibration
Status - Whether calibration may recommend a threshold from the available evidence.
- Evaluation
Error - Content-free validation errors for evaluation artifacts.
- Expected
Answer - Independently labeled answer for one fixture question.
- Failure
Kind - Safe failure categories permitted in saved results.
- Saved
Outcome - Explicit success or failure, without raw provider errors or request content.
Constants§
- EVALUATION_
SCHEMA_ VERSION - Current fixture, saved-response, and report schema version.
- MAX_
EVALUATION_ CASES - Maximum number of cases in one evaluation suite.
Functions§
- calibrate
- Select a question-specific threshold only when the 95% lower confidence bound meets the requested accepted-accuracy target on enough examples.
- compare
- Compare two saved reports without turning a regression into a CI gate.
- evaluate
- Evaluate saved responses without network access or provider credentials.