Skip to main content

Module evaluation

Module evaluation 

Source
Expand description

Versioned, provider-independent offline decision evaluation.

Structs§

AcceptancePoint
Accuracy and coverage at one observed confidence cutoff.
CalibrationResult
Per-question calibration suggestion, never applied to production configuration.
ComparisonReport
Read-only comparison result; regressions do not change the process exit code.
Completeness
Completeness counts; failures and missing records never count as incorrect predictions.
EvaluationReport
Deterministic offline evaluation report.
FixtureCase
A labeled request, with language, risk, and provenance for grouped analysis.
FixtureSuite
A versioned set of labeled requests. It contains no provider credentials.
MetricRegression
One changed metric in a comparable evaluation pair.
QuestionMetrics
Metrics for one question and optional language/risk slice.
SavedResponses
Versioned saved provider results for fully offline evaluation.
SavedResult
One saved case result.

Enums§

CalibrationStatus
Whether calibration may recommend a threshold from the available evidence.
EvaluationError
Content-free validation errors for evaluation artifacts.
ExpectedAnswer
Independently labeled answer for one fixture question.
FailureKind
Safe failure categories permitted in saved results.
SavedOutcome
Explicit success or failure, without raw provider errors or request content.

Constants§

EVALUATION_SCHEMA_VERSION
Current fixture, saved-response, and report schema version.
MAX_EVALUATION_CASES
Maximum number of cases in one evaluation suite.

Functions§

calibrate
Select a question-specific threshold only when the 95% lower confidence bound meets the requested accepted-accuracy target on enough examples.
compare
Compare two saved reports without turning a regression into a CI gate.
evaluate
Evaluate saved responses without network access or provider credentials.