Module ai_evals_api 

Enums

CollectExperimentError

struct for typed errors of method collect_experiment

CreateEvalDatasetError

struct for typed errors of method create_eval_dataset

DeleteEvalDatasetError

struct for typed errors of method delete_eval_dataset

EvalRunPayloadError

struct for typed errors of method eval_run_payload

EvalSubjectStateError

struct for typed errors of method eval_subject_state

ExperimentResultsError

struct for typed errors of method experiment_results

GetEvalDatasetError

struct for typed errors of method get_eval_dataset

ListAllExperimentsError

struct for typed errors of method list_all_experiments

ListEvalCasesError

struct for typed errors of method list_eval_cases

ListEvalDatasetsError

struct for typed errors of method list_eval_datasets

RecentScorersError

struct for typed errors of method recent_scorers

RunExperimentError

struct for typed errors of method run_experiment

ScorerDefaultsError

struct for typed errors of method scorer_defaults

UpdateEvalDatasetError

struct for typed errors of method update_eval_dataset

Functions

collect_experiment

Called by a run’s own flow as its last step. The answers and scores a run produced live in its jobs, which have their own retention; this copies them onto the run’s rows. Reading a run does the same, so this is what covers a run nobody opened.

create_eval_dataset

delete_eval_dataset

The cases, the runs and their recorded case sets go with it through the foreign keys; the jobs those runs produced are left alone.

eval_run_payload

Called by the step a run’s flow places between the agent and its scorers. Every tool call is enriched with the arguments, result, status and duration of the job that ran it, and with the schema of the script version it ran, none of which the flow itself can read.

eval_subject_state

The version it is deployed at. Small on purpose: the results endpoint reports the same thing but harvests scores and reads every job to do it, so it is not something to ask for on its own.

experiment_results

get_eval_dataset

list_all_experiments

list_eval_cases

list_eval_datasets

recent_scorers

Filtered twice, both times by what the caller can read: the datasets they come from, and the runnables themselves. A scorer they could not run does not appear.

run_experiment

scorer_defaults

update_eval_dataset