Expand description
Deterministic helpers for testing Runifold runtime semantics.
Structs§
- Evaluation
Case - One immutable evaluation input and its optional reference answer.
- Evaluation
Case Id - Stable case identity within a dataset.
- Evaluation
Case Result - Output-free per-case evaluation result.
- Evaluation
Dataset - Versioned, duplicate-free evaluation dataset.
- Evaluation
Failure - Safe per-case execution or scorer failure.
- Evaluation
Metrics - Non-sensitive resource evidence for one successful target execution.
- Evaluation
Output - Target output plus optional Run/Trace correlation.
- Evaluation
Report - Deterministic candidate evaluation report.
- Evaluation
Runner - Concurrent deterministic evaluation orchestrator.
- Evaluation
Score - One persisted per-case score.
- Evaluation
Score Summary - Aggregate score statistics.
- Fault
Controller - Shared deterministic fault plan and execution counters.
- Fault
Injecting Model - Model wrapper applying one queued fault per invocation.
- Fault
Injecting Tool - Tool wrapper applying named, one-based invocation faults.
- File
Evaluation Repository - Filesystem-backed immutable JSON evaluation repository.
- FnScorer
- Closure-backed asynchronous scorer.
- Golden
Trace - Versioned, serializable execution trace for behavioral regression tests.
- Json
Exact Match Scorer - Deterministic JSON equality scorer.
- Json
Rule Scorer - Weighted deterministic scorer for structured outputs.
- Judge
Rubric - Versioned rubric for a structured model judge.
- Metric
Regression - One baseline-to-candidate score comparison.
- Model
Judge Scorer - Canonical-model-backed structured LLM judge.
- Model
Scenario - Accumulates a scripted provider event sequence without network access.
- Recovery
Harness - Reconstructs runtime state from one factory while preserving external fixtures such as stores, fault counters, and cassette servers.
- Regression
Comparison - Complete relative regression decision.
- Regression
Policy - Allowed relative quality drops.
- Retrieval
Case Metrics - Metrics for one ranked retrieval case.
- Retrieval
Evaluation Case - One versionable retrieval-quality case.
- Retrieval
Evaluation Report - Macro-averaged retrieval-quality report.
- Retrieval
Evaluation Runner - Deterministic evaluator for any provider-neutral retriever.
- RunScenario
- An isolated root run with deterministic event collection.
- Score
Value - Validated score value and optional evaluator rationale.
- Scripted
Model - A deterministic, queue-backed model adapter.
- Token
Overlap Scorer - Deterministic lexical overlap scorer for string reference answers.
- Tool
Fault - Fault applied to one named Tool invocation.
- Weighted
Json Rule - One weighted deterministic JSON-output rule.
Enums§
- Evaluation
Error - Evaluation configuration or execution failure.
- Evaluation
Failure Stage - Evaluation failure stage.
- Evaluation
Store Error - Evaluation artifact repository failure.
- Golden
Trace Mismatch - Typed golden-trace comparison failure.
- Json
Rule - Deterministic rule over a target JSON output.
- Model
Fault - Fault applied to one model invocation.
- Retrieval
Evaluation Error - Invalid dataset or failed retrieval evaluation.
- Scenario
Assertion Error - Typed failure from a scenario assertion.
Traits§
- Evaluation
Repository - Versioned dataset and candidate-report repository.
- Evaluation
Scorer - Asynchronous scorer boundary.
- Evaluation
Target - Asynchronous system-under-evaluation boundary.
Type Aliases§
- Evaluation
Future - Owned asynchronous evaluation operation.
- Fault
Scenario - Fluent scenario name for
FaultController.