Module verify 

Workload verification — run a workload and check its output against rules embedded in the workload file. Shared by the nmbrs check subcommand and the example-walker test, so “how CI checks the examples” and “how a user checks their own workload” are the same code.

A verification target is resolved the same way nmbrs run resolves workload=…: a directory (walk every workload under it), an existing .yaml/.yml file (run it by path), or a bundled catalog name such as examples/cursors/all_cursor/enumerate (run it by name, read its rules from the embedded source). So whatever tab-completion offers for nmbrs check <TAB> — local files and catalog names — checks the same way it runs.

Two equivalent rule surfaces (a file may use either, or both — their cases combine):

1. #@ comment directives — trailing YAML comments, inert to the runtime:

   #@ run scenario=enumerate
   #@ expect 50 completed, 0 failed
   #@ case overload
   #@   run concurrency=32 rate=100000
   #@   expect-fail error_rate_exceeded
   #@ requires backend (needs a live service)
   #@ session cwd            (sessions under the sandbox cwd — stick_session)
   #@ again phases=probe     (a second invocation, session state preserved)

2. A verify: YAML block — also inert (the runtime ignores unknown top-level keys). Three equivalent shapes:

   # single case (a directive map)
   verify: { run: scenario=enumerate, expect: "50 completed, 0 failed" }
   # a list of cases
   verify:
     - { case: a, run: scenario=a, expect: "..." }
     - { case: b, expect-fail: "..." }
   # a name-keyed map (key = case name)
   verify:
     a: { run: scenario=a, expect: "..." }
     b: { expect: ["x", "y"] }

expect / expect-fail accept a single regex or a list of regexes. Each must match the run’s combined stdout+stderr; expect-fail additionally requires a non-zero exit.

Structs

VerifyCase

One verification case: an invocation plus the regexes its output must match.

VerifyPlan

The parsed verification plan for one workload file.

VerifySummary

Aggregate verification result across one or more files.

WorkloadTiming

Wall-clock spent checking one workload, with its aggregate status.

Enums

CheckProgress

Live verification progress, emitted as workloads start and finish so a caller (e.g. nmbrs check) can render an active/pending/done/errors status line. Invoked from worker threads — handlers must be Sync.

CheckStatus

A single workload’s aggregate outcome, for live progress and the timing report. Coarser than Outcome (which is per case): a workload with any failing case is CheckStatus::Fail; one with no failures and at least one skip (and no pass) is CheckStatus::Skip; otherwise CheckStatus::Pass.

Outcome

What one verification case produced.

WorkloadSource

Where a verification target’s rule text and run reference come from. Mirrors nmbrs run’s workload=… resolution.

Constants

DEFAULT_TIMEOUT_SECS

Default per-case run timeout, in seconds.

Functions

check_case_output

Check one case’s RUN RESULT against its expectations — the run-mechanism- agnostic half of run_case. combined is the merged stdout+stderr the run produced; succeeded is its exit success; timed_out is the timeout signal. Used both by the subprocess run_case and by nmbrs’s in-process run_executions-backed verification (so the SAME expect / expect-fail rules apply whether examples run as subprocesses or as concurrent in-process executions sharing one session).

collect_workload_files

Every *.yaml / *.yml workload file under dir (recursive), sorted. The discovery half of verify_path, exposed so the in-process example walker (nmbrs_runtime::verify_in_process) finds the same files the subprocess walker does.

declared_params

A workload reference’s declared top-level params: (string scalars), following its extends: chain — resolved the way nmbrs run resolves workload=…. Returns None if the reference resolves to nothing or has no params: block.

no_progress

No-op progress handler, for callers that only want the summary.

requires_verification_rules

Whether a workload is REQUIRED to declare verification rules.

resolve_ref

Resolve a single workload reference the way nmbrs run does: an existing .yaml/.yml file path first, then a bundled catalog name. None if it is neither. Directories are not a single workload — callers handle those separately (via verify_path).

run_case

Run one case via <binary> run workload=… <args> --session-path … (under an in-process deadline) from sandbox, capture combined output, and check the rules. workload_ref is whatever goes after workload= — an absolute file path or a bundled catalog name; the subprocess resolves it exactly as a normal nmbrs run would.

verify_file

Verify one workload file: read it, then run it by its absolute path. The file is read relative to this process’s cwd, but the run reference is made absolute because each case is launched from a sandbox cwd.

verify_path

Verify a workload file or every *.yaml under a directory (recursively). Files run concurrently (cases within a file are sequential). progress is invoked from worker threads as each workload starts and finishes — pass no_progress for a quiet run.

verify_source

Verify a workload from its rule text + run reference: parse the rules, run every case, return one (label, Outcome) per case (or a single Skip / Fail for the whole workload). run_ref is what to pass as workload= — an absolute file path or a catalog name.

verify_target

Verify a target named the way nmbrs run names workloads: a directory (walk every workload under it), an existing workload file, or a bundled catalog name (examples/cursors/all_cursor/enumerate, …). This is the nmbrs check entry point — so anything the binary can run by name, it can check by the same name.

Type Aliases

ProgressFn

A progress handler. &-shared across verification worker threads.