Module verify 

Verify - the Phase-3 live conformance smoke suite, sanitized failure evidence, and the versioned model-health.json store (task sa-0).

Three cheap, non-destructive checks exercise the OpenAI-compatible Chat Completions surface exactly as the Hermes plugin uses it:

1. text_completion - a non-streaming completion must return the exact string CF_HERMES_OK in under 30 seconds (feedback 02 10-inference acceptance: content_exact + max_elapsed_ms).
2. streaming - a streaming completion must emit its first valid SSE event within 20 seconds, deliver at least one content delta, reach a terminal chunk (finish_reason non-null), close without a protocol error, and never emit a duplicate terminal event (feedback 02 20-streaming acceptance). SSE is hand-rolled (line-based data: parsing) because the workspace forbids extra dependencies.
3. tool_call - a single call to the harmless get_project_sentinel function with scope=provider-conformance exactly once (feedback 02 30-tool-calling acceptance).
4. structured_output - a single JSON object with sentinel exactly CF_HERMES_OK, constrained prompt-only (feedback 02 50-structured-output acceptance).
5. parallel_tools - two distinct harmless tool calls in one assistant turn (feedback 02 30-tool-calling acceptance).

Checks 4 and 5 are standalone: they are exposed as public functions and are deliberately NOT part of run_smoke_suite (wiring them into the suite or the CLI is a separate task).

Live gate: every HTTP-calling function refuses to run unless the environment variable AUTH_CLOUDFLARE_LIVE_TESTS is exactly 1 - live_tests_enabled() - returning a typed CloudflareError::MissingEnv refusal that names the variable. This is the guardrail against accidental paid inference (feedback 02: paid runs are opt-in). AUTH_CLOUDFLARE_MAX_COST_USD is an optional documented conformance budget: when present, the run report carries cost_estimate_usd; the value is reported and documented but never enforced - enforcement is the operator’s job.

Exit-code contract (feedback 02, binding table):

• 0 - all three checks passed;
• 1 - the live gate is closed (operational refusal);
• 2 - credentials missing (resolved by the CLI before this module);
• 3 - the live API was unreachable at send time (no HTTP response was obtained - connect/DNS/TLS/overall-timeout), i.e. a remote Cloudflare API failure, not a conformance verdict;
• 6 - the suite ran to completion and at least one check failed acceptance (non-2xx status or a 200 that failed acceptance), with sanitized FailureEvidence persisted.

Security contract (feedback 02, CI-tested): the token travels exclusively through crate::fetch::auth_header; every error text and every response excerpt is scrubbed with redact_token; FailureEvidence::new caps excerpts at 512 characters; prompts, tool outputs, Authorization headers and tokens are never persisted. The health store records only sanitized evidence.

Health store layout (under Config::cache_dir()): model-health.json - { version: 1, updated_at, records: { <model_id>: ModelVerification } }, written atomically (sibling .tmp + rename, 0o600) via crate::cache::atomic_write.

Structs

CheckOutcome

Outcome of one check: passed + elapsed, with sanitized failure evidence when the check failed.

ChecksReport

The three smoke checks, keyed for the CLI report.

HealthStore

The versioned health store: one ModelVerification per model id, keyed by model_id. Lives at Config::cache_dir()/model-health.json, written atomically (sibling .tmp + rename, 0o600).

SmokeRunReport

Full smoke run report (the CLI adds gate and exit_code).

Enums

StreamEvent

One parsed SSE event: a JSON data: payload, the [DONE] sentinel, or a malformed data: line (a protocol error).

SuiteKind

The suites this runner knows. smoke and tool-loop exist in Phase 3; the CLI rejects anything else as a usage error.

Constants

EXACT_TEXT

The exact content the non-streaming check must deliver.

HEALTH_STORE_FILE

File name of the health store, inside Config::cache_dir().

HEALTH_STORE_VERSION

Schema version of the health store file layout. Bump on layout change so older stores are rejected loudly instead of misread.

LIVE_TESTS_ENV

Env var gating live inference: the suite runs only when this is exactly "1" (feedback 02: paid runs are opt-in).

MAX_COST_ENV

Optional conformance budget env var. When present (a positive finite number), the run report carries cost_estimate_usd; the budget is documented and reported, never enforced.

PARALLEL_TOOL_CHECK_TIMEOUT

Overall agent timeout for the parallel-tools check.

PARALLEL_TOOL_NAME

The second harmless sentinel tool name (parallel-tools check).

PARALLEL_TOOL_PROMPT

The parallel-tools check prompt (feedback 02 30-tool-calling): both harmless tools must be called in one assistant turn.

PARALLEL_TOOL_VALUE

The only permitted marker argument value for the second tool.

SMOKE_SUITE_ESTIMATED_COST_USD

Conservative upper-bound cost estimate for one smoke run (three tiny requests, roughly 80 tokens round-trip at the most expensive in-catalog price). Reported as cost_estimate_usd only when AUTH_CLOUDFLARE_MAX_COST_USD is set; the budget itself is documented, never enforced.

STREAM_CHECK_TIMEOUT

Overall agent timeout for the streaming check - the “configured timeout” capping total stream duration.

STREAM_FIRST_EVENT_MAX_MS

Streaming acceptance: first valid SSE event within this many ms.

STREAM_FIRST_EVENT_TIMEOUT

Per-read timeout for the streaming check (a stalled upstream must not hold the first event past the 20 s acceptance window).

STRUCTURED_CHECK_TIMEOUT

Overall agent timeout for the structured-output check.

STRUCTURED_PROMPT

The structured-output check prompt. Prompt-only JSON constraint (no response_format parameter) so models that reject that parameter as unsupported still exercise structured output.

STRUCTURED_SENTINEL_FIELD

The required field name in the structured-output object.

STRUCTURED_SENTINEL_VALUE

The required field value in the structured-output object.

TEXT_CHECK_TIMEOUT

Overall agent timeout for the non-streaming text check (enforces the 30 s acceptance at the transport level).

TEXT_COMPLETION_MAX_ELAPSED_MS

Non-streaming acceptance: elapsed_ms must stay under this (feedback 02).

TEXT_PROMPT

The fixed user message for the text and streaming checks (feedback 02 10-inference).

TOOL_CHECK_TIMEOUT

Overall agent timeout for the tool-call check.

TOOL_NAME

The harmless sentinel tool name (feedback 02 30-tool-calling).

TOOL_PROMPT

The tool-calling check prompt (feedback 02 30-tool-calling).

TOOL_SCOPE

The only permitted scope argument value.

Functions

aggregate_verification

Aggregate the three check outcomes into the canonical ModelVerification (feedback 02). Status formula: 0 failures → Passing, 1 failure → Degraded, 2+ failures → Failing. Each check counts as one run (total_runs = 3), so delivery_success_rate() is the fraction of smoke checks that delivered - the picker’s delivery metric. last_failure is the first failed check’s evidence (text, stream, tool order); failure counters map from FailureClass.

check_parallel_tools

Check (e): parallel tool calls. Acceptance (feedback 02 30-tool-calling): exactly two tool calls in one assistant turn, one get_project_sentinel and one get_project_marker (order-agnostic), both with valid JSON arguments.

check_streaming_completion

Check (b): streaming completion. Acceptance (feedback 02 20-streaming): first valid SSE event within 20 s, at least one content delta, a terminal chunk with non-null finish_reason, clean close with no protocol error, no duplicate terminal event, total duration under the configured timeout.

check_structured_output

Check (d): structured output. Acceptance (feedback 02 50-structured-output): the completion content parses as a JSON object with sentinel exactly CF_HERMES_OK. Prompt-only constraint - no response_format parameter, so the model must honor a natural-language shape instruction.

check_text_completion

Check (a): non-streaming exact-text completion. Acceptance: content exactly CF_HERMES_OK, elapsed under 30 000 ms.

check_tool_call

Check (c): one harmless tool call. Acceptance (feedback 02 30-tool-calling): exactly one standard tool call, name exactly get_project_sentinel, argument JSON parses, scope exactly provider-conformance.

classify_completion_failure

Classify a non-streaming completion body that failed acceptance: empty content → EmptyCompletion, truncated (finish_reason “length”) → TruncatedCompletion, invalid JSON → InvalidJson, wrong shape → InvalidChatCompletionShape, missing/null finish reason → MissingFinishReason, else Unknown.

classify_http_failure

Classify a non-2xx (or envelope-carrying 200) response: 401 → AuthRejected, 403 → AuthRejected or AccountNotFound per the Cloudflare error envelope, 429 → RateLimited, 5xx → ProviderServerError, and an envelope with success: false plus a cf-ray → CloudflareEdgeError. Everything else is Unknown - never guess a specific class without evidence.

classify_parallel_tools

Pure acceptance for the parallel-tools check (feedback 02 30-tool-calling): exactly two tool calls in one turn, one per expected name (get_project_sentinel + get_project_marker, order-agnostic), both with valid JSON arguments. None means acceptance holds; otherwise NoToolCall (fewer than two calls), DuplicateToolCall (more than two, or the same name twice), InvalidToolName (an unexpected name), or InvalidToolArguments (an argument string that is not valid JSON).

classify_stream_failure

Pure streaming acceptance over a fully read event stream. None means every acceptance criterion holds: first valid event within 20 s, at least one content delta, a terminal chunk with a finish reason, no protocol error, total duration under the configured timeout.

classify_structured_output

Pure acceptance for the structured-output check (feedback 02 50-structured-output): the completion content must parse as a JSON object with sentinel exactly CF_HERMES_OK. None means the criterion holds; otherwise the closest existing taxonomy class is returned (InvalidJson for unparsable/wrong-value JSON, InvalidChatCompletionShape for a non-object, EmptyCompletion for empty content) - no new variant is introduced.

classify_tool_response

Classify a tool-call response that failed acceptance: no tool call → NoToolCall, wrong name → InvalidToolName, unparsable/wrong arguments → InvalidToolArguments, more than one call → DuplicateToolCall, malformed envelope → InvalidJson/InvalidChatCompletionShape. None means the response is a valid single get_project_sentinel(scope=provider-conformance) call - FailureClass::Unknown must never be conflated with “valid”.

classify_transport_message

Classify a ureq transport message into the failure taxonomy by keyword: timeouts → ReadTimeout, connection failures → ConnectTimeout, resets → ConnectionReset, TLS → TlsFailure, else Unknown.

gate_refusal

The typed refusal returned when the live gate is closed.

inspect_text_completion

Pure acceptance for the non-streaming text check: content must be exactly CF_HERMES_OK in a well-formed completion. Returns the failure class, or Ok(()) when the content criterion holds (latency is checked by the caller).

live_tests_enabled

True when the live gate is open: AUTH_CLOUDFLARE_LIVE_TESTS is exactly "1". Pure and testable; every HTTP-calling function consults this first.

load_health_store

Load the health store from dir/model-health.json. A missing file is an empty store; a present-but-corrupt file or an unsupported schema version is a typed error (never a panic).

max_cost_usd_env

Optional AUTH_CLOUDFLARE_MAX_COST_USD budget: a positive finite number, when present. Reported as cost_estimate_usd; never enforced.

run_smoke_suite

Run the full smoke suite against the model id exactly as given - the catalog is never consulted, so a model absent from the catalog is still verified (the CLI contract: fall back to the id as passed).

save_health_store

Persist the health store atomically under dir (create dirs, sibling .tmp + rename, 0o600). Never leaves a half-written store behind.

save_verification

Read-modify-write upsert of one verification record after a run: load the existing store (preserving every other model’s record), upsert, save.

verification_from_tool_loop

Aggregate one tool-loop outcome into a ModelVerification (feedback 02 suite 40). The loop counts as one run: total_runs = 1, multi_turn_tool_success_rate is 1.0 when the loop converged, 0.0 otherwise; a non-converged loop is Failing with tool_loop_failures = 1 and the loop’s own FailureClass as sanitized evidence (no HTTP status, no excerpt - the loop carries no response body).