Crate systemprompt_evaluation 

Evaluation domain crate for systemprompt.io.

Closes the loop on the platform’s AI request trace (ai_requests and its satellite tables, owned by systemprompt-ai): samples production traffic, scores it against rubrics with an LLM judge, and replays failures with a repair hint so repaired trajectories are scored and linked to the results they fix.

Public surface is a typed EvaluationError boundary, repositories over the eval_* tables, and the services composing them: EvaluationService, SamplerService, JudgeService, ReplayService, and AutoImproveLoop. Judge and replay inference go through a DynAiProvider supplied by the composition layer; it must be the platform’s auditing implementation, which persists every request to ai_requests.

Judge and replay requests are attributed with a job actor so sampling — which excludes actor_kind = 'job' — never grades the framework’s own traffic.

Copyright (c) systemprompt.io — Business Source License 1.1. See https://systemprompt.io for licensing details.

Re-exports

pub use error::EvaluationError;

pub use error::Result;

pub use extension::EvaluationExtension;

pub use models::CanonicalMessage;

pub use models::CanonicalPrompt;

pub use models::DimensionScore;

pub use models::EvalCase;

pub use models::EvalResult;

pub use models::EvalRun;

pub use models::EvalRunKind;

pub use models::EvalRunStatus;

pub use models::JudgeVerdict;

pub use models::NewCaseParams;

pub use models::NewResultParams;

pub use models::NewRunParams;

pub use models::Rubric;

pub use models::RubricDimension;

pub use models::SampleFilter;

pub use models::SampledRequest;

pub use models::TriggerSource;

pub use models::Verdict;

pub use repository::EvalCaseRepository;

pub use repository::EvalJudgeCallRepository;

pub use repository::EvalRepositories;

pub use repository::EvalResultRepository;

pub use repository::EvalRubricRepository;

pub use repository::EvalRunRepository;

pub use repository::SamplingRepository;

pub use services::AutoImproveLoop;

pub use services::EvaluationService;

pub use services::JudgeService;

pub use services::LoopLimits;

pub use services::LoopReport;

pub use services::ReplayService;

pub use services::RunRequest;

pub use services::SamplerService;

Modules

error

Typed error boundary for the systemprompt-evaluation crate.

extension

Extension registration — wires the evaluation schemas (runs, cases, results, pairs, judge calls, rubrics) and their reconcile migrations into the extension framework.

models

Data model for evaluation runs, cases, results, rubrics, and sampling.

repository

Repositories over the eval_* tables plus the sampling reader over the ai_requests trace owned by systemprompt-ai.

services

Evaluation services: sampling, judging, replay, and the auto-improve loop.