Module eval_cmd 

talon eval score and talon eval run: the golden-set regression harness for investigation quality.

The scoring itself is pure and lives in talon_intel::eval; this module does the I/O around it:

• loading golden cases (evals/golden/<case id>.json),
• score: reading one already-produced report per case from a directory and printing the summary,
• run: submitting every case to a running Talon through the CoPilot HTTP API (POST /investigate, then polling GET /investigations/{id}), i.e. the same path CoPilot uses, and scoring the verdict Talon recorded.

Reports are agent output: they are read without following symlinks and with a size cap (crate::safe_fs), parsed as data only, and whatever of them reaches the terminal has been stripped of control characters by the scorer.

Exit codes: 0 pass rate reached, 1 pass rate below --min-pass-rate, 2 unusable input or configuration.

Structs

CaseRun

What investigating one case through the API produced.

HttpInvestigationApi

InvestigationApi over HTTP against a running Talon.

RunOptions

How the run loop waits.

Submission

Answer of POST /investigate.

Constants

DEFAULT_CASES_DIR

Golden-case directory below the project root.

EXIT_BELOW_THRESHOLD

Exit code: the pass rate is below the threshold.

EXIT_USAGE

Exit code: unusable input or configuration.

MAX_CASE_BYTES

Largest golden-case file read.

MAX_POLL_ERRORS

Consecutive failed status polls after which a case is given up.

MAX_REPORT_BYTES

Largest report / investigation record read (file or HTTP body).

TRIGGERED_BY

triggered_by of harness-submitted investigations (the HTTP API accepts manual, scheduled and webhook).

Traits

InvestigationApi

The slice of the CoPilot HTTP API the harness uses; a trait so the run loop is tested without a server.

Functions

investigate_body

The POST /investigate body for a case request.

investigate_case

Submit request and wait for its investigation to finish.

load_cases

Load and validate every *.json golden case in dir, sorted by file name. The file stem must equal the case id.

load_outcome

What <case_id>.json in the open reports directory says about a case.

parse_alert_map

Parse an --alert-map document: {"<case id>": <alert id>, ...}.

prepare_requests

The request to submit for each case: the case’s own, with the alert id from alert_map and the customer code from customer_code when given. Validated against the HTTP API’s input rules before anything is sent.

run

talon eval run: returns the process exit code.

run_cases

Investigate every case in turn (one at a time: the investigation lane is serialized anyway, and a per-case timeout stays meaningful) and score the outcomes. progress receives one line per step.

score

talon eval score: returns the process exit code.

score_reports

Score the reports in reports_dir against cases.