Expand description
Tool mocking and spying: the mocks: block of a test case (and the CLI’s
--mocks file), compiled down to the oneharness mock ruleset, plus the
MockCall records that come back — one per tool call the harness’s hook
observed, carrying the original pre-rewrite input and the verdict applied.
Vocabulary (sinon’s): a spy observes without intercepting (a declaration
with no action), a stub substitutes a canned shell result, deny
blocks with a model-visible message, and rewrite substitutes raw input
fields (the primitive under stub, and the way to mock file reads). First
matching action wins; a call no action-rule matches is allowed through and
still recorded.
Two matching engines exist on purpose:
- Action rules are matched inside the harness’s hook process by
oneharness mock— [compile_rules] renders that ruleset. skilltest mirrors the same semantics indecideso the bundled fake provider (and the gate) exercise the identical decision logic without a harness; the live per-harness e2e is the drift alarm between the two. - Spy declarations and eval
whereclauses are matched locally over the returned records (decl_matches/where_matches).
Everything here is validated loudly at load time — an invalid regex, an empty needle, or a criterion-less matcher must abort the run, never degrade to match-nothing (or match-everything).
Structs§
- Mock
Call - One observed tool call, as recorded by the mock/spy channel: the harness
hook’s spy log for real runs, or the provider’s
mock_callsresponse for the command protocol. Carries the original, pre-rewrite input — the transcript’seventsshow post-rewrite reality (the stub that actually ran); this shows what the skill attempted. - Mock
Decl - One mock or spy declaration — an entry of a test case’s
mocks:block, of the CLI’s--mocksfile, or synthesized by an SDK. Exactly one ofstub/deny/rewritemakes it a mock (the call is intercepted); none makes it a spy (observed only, matched locally against the returned records). - Mock
Match - What a mock/spy declaration matches on. At least one criterion is required; every given criterion must hold (AND).
- Mock
Plan - What the provider should do about mocking for one
respondcall: the compiled ruleset to enforce (if any) and that the observation channel is on. Absent entirely (the runner passesNone) when the case declares nothing. - MockSet
- The effective mock/spy set for one case run: the CLI/SDK-level declarations (first, so a test-local rule shadows a shared one — first match wins) followed by the case’s own, validated as a whole and compiled to the oneharness ruleset.
Enums§
- Applied
Action - A matched action, borrowed from the compiled ruleset.
- Deny
Spec - A
denyaction: block the call with a model-visible message. Written as a bare string (the message) or a map withmessage. - Field
Predicate - A predicate on one tool-input field. Written in YAML either as a bare
string (exact equality) or as a map with any of
equals/contains/pattern(all given forms must hold). - Stub
Spec - A
stubaction: fake a shell call’s result by declaring only the output. Written as a bare string (the output) or a map withoutput+exit_code.
Functions§
- decide
- Decide which rule of a compiled ruleset (the
{"rules": […]}value built byMockSet::rules) intercepts a call. First match wins;Nonemeans allow through. This mirrorsoneharness mock’s decision over the same shape so the fake provider exercises identical semantics. - decl_
matches - Whether a spy declaration’s matcher covers a record (local matching — the hook never sees spies).
- describe_
records - A compact one-line description of observed calls for failure messages,
e.g.
bash(git status), read({"file_path":"x"}) [deny]— capped so a chatty run doesn’t flood the report. - parse_
spy_ log - Parse an oneharness spy-log (JSONL) into records. Unparseable lines are an error — a truncated log must not silently read as “fewer calls”.
- stub_
command - The shell command a
stubcompiles to — a safely single-quotedprintfof the declared output, exactly mirroring oneharness’sstub_inputso the fake provider’s post-rewriteeventslook like the real thing. - validate_
where - Validate an eval
whereclause, identified aswhoin errors. - where_
matches - Apply an eval’s
whereclause (per-field input predicates) to a call’s input. Every listed field must exist and match; an empty clause matches.