sharpebench-protocol 0.21.0

SharpeBench language-agnostic agent ⇄ harness protocol (JSON message types).
Documentation

The language-agnostic agent ⇄ harness protocol.

Agents are external — a container or HTTP endpoint, in any language — not Rust code. Each decision step the harness sends a [MarketObservation] (JSON) and the agent replies with a [Decision] (JSON). Keeping this surface tiny and stable is what lets any vendor compete (and is the whole adoption story).

All observations are point-in-time: close_history, fundamentals and news only ever contain information available at or before date.

The wire contract is closed (breaking for entrants as of 0.11.0)

Every wire type carries #[serde(deny_unknown_fields)]. An agent that emits a key the contract does not define is rejected at the transport boundary and scored as an agent protocol fault, not silently accepted. This is a deliberate departure from the additive-only discipline the rest of the artifact formats follow: an attested benchmark cannot let an unread field carry meaning the scorer never saw.

The authoritative machine-readable definition of the closed contract is published as JSON Schema (draft 2020-12) alongside this crate: schema/decision.schema.json and schema/observation.schema.json. Both set additionalProperties: false to mirror deny_unknown_fields, and a bidirectional drift guard (tests/schema_drift.rs) fails the build if the schema and the Rust types disagree in either direction.

Entrants migrating from 0.10.x: drop any extra keys, or move them under reasoning (free text) or cost (structured spend). [decision_from_wire] produces the diagnostic that names the offending field.

Decisions must be deterministic under re-execution

A run may be executed more than once: a runtime failure is retried by restarting the run from its first step with a fresh agent, an interrupted sweep resumes by running its unfinished runs again, and a verifier may re-execute a captured trajectory to check it. Each decision must therefore be a deterministic function of the observations of the same run up to and including the one being answered, and of the agent's own earlier decisions in that run. Behaviour that depends on anything else (wall-clock time, ambient randomness, or state carried in from another run or from outside the run) may diverge on re-execution, and re-execution verification refuses a run that diverges.

What must repeat is the score-bearing decision: every order's symbol, action, target_weight and confidence, and the cost report. reasoning and each order's rationale are audit text that the scorer never reads, so they may differ. An agent that wants randomness must derive it from the observation stream it was given.

The harness checks this in sharpebench_harness::verify_trajectory_reexecuted: after the strict artifact checks pass, every captured run is re-executed with a fresh agent on the same frozen data, window and seed, and the first decision that differs is refused as a typed ReexecutionDivergence naming the run, the step and the observation, never accepted as a silently different run. Replaying recorded decisions alone (sharpebench_harness::verify_trajectory_strict) is exact by construction and cannot see a non-deterministic agent; re-execution is the check that can. A sweep's own retries and resumes do not compare a rerun against the attempt it replaced, so a non-deterministic agent is caught when its trajectory is re-executed, not while the sweep runs. The command line exposes the check as sharpebench verify-trajectory --reexecute.

Consistency relaxations a fault plan may declare

An operator may run an entrant under a seeded fault plan (sharpebench run --fault-plan, sharpebench_harness::fault_plan). A plan must declare exactly the relaxations its faults use, or it is refused, and the operator publishes that declaration to the entrant before the sweep. A faulted observation may violate only a declared relaxation, only for a bounded number of decision steps or presentations (at most 64), and never in the book: every fault changes what the entrant is shown or which of its submissions is accepted, never the executed book or the returns the scorer reads. The wire shape is the same under every relaxation, and without a plan none of them applies. Each relaxation is named here by its wire name in the plan's declared_relaxations:

  • read_your_writes: after an order executes, cash and portfolio on later observations may show their pre-execution values for a bounded number of decision steps. The order has executed and the book is authoritative; date and symbols stay current. The harness records convergence when an observation is seen to carry the canonical holdings again, not when a timer expires. Restating the last target is the idempotent response; pushing the target further in the direction of the hidden write is graded as escalation.
  • position_sign_convention: a nonzero portfolio[].shares may be shown with the opposite sign for a bounded number of decision steps. Zero has no sign and is left alone, and the book is unchanged. The response is graded against the entrant's own last stated target for the symbol.
  • submission_acceptance: a decision that carries orders may be rejected under a rate limit. Rejection is signalled by presenting the identical observation again, a bounded number of times; only the decision answering the first presentation after that deadline executes. A hold is never rejected. A deterministic entrant restates its decision and loses nothing.
  • complete_results: declared for completeness only. No plan can arm it, because the observation contract has no paged read.

The grades of the entrant's responses are rank-neutral evidence on the attempt ledger. They never enter a return, a score or a rank.

The declared metadata of each wire operation (does it change state, is it safe to repeat, may the harness retry it) is published beside the schema; see [operations].