Crate cellos_fleet

Expand description

cellos-fleet — host-resident agent that polls a spec queue and dispatches execution cells to cellos-supervisor.

§Library seam (S24)

This crate exposes a thin library target alongside its binary so that the configuration surface (Config) and node-identity surface (NodeIdentity) can be exercised by integration tests and reused by future control-plane tooling without shelling out to the binary. The binary (src/main.rs) is a thin wrapper: it parses --version, installs the tracing subscriber, builds a Config via Config::from_env, and drives run. No runtime behaviour changed when the seam was carved — every item below was lifted verbatim from the original main.rs.

§Queue model

The agent treats an S3 prefix as a simple work queue using key renaming as the claim primitive:

pending/<spec-id>.json   →  claimed/<spec-id>.json  →  supervisor runs
                                                     →  completed/<spec-id>.json  (exit 0)
                                                     →  failed/<spec-id>.json     (exit ≠ 0)

Claiming is performed by aws s3 mv (copy + delete), which is not atomic but is safe enough for low-concurrency single-agent deployments. A future version can replace this with a DynamoDB conditional write for true atomic claim.

§Environment variables

Variable	Required	Description
`CELLOS_FLEET_BUCKET`	yes	S3 bucket name
`CELLOS_FLEET_PREFIX`	no	Key prefix inside bucket (default: `fleet`)
`CELLOS_FLEET_QUEUE_NAME`	no	Optional queue lane under the prefix
`CELLOS_FLEET_POOL_ID`	no	Runner pool identifier; T11 placement gate. When set, the dispatcher skips specs whose `spec.placement.poolId` is set AND does not equal this value. Specs without a `poolId` constraint are accepted everywhere.
`CELLOS_FLEET_SUPERVISOR`	no	Path to cellos-supervisor binary (default: `cellos-supervisor`)
`CELLOS_FLEET_POLL_INTERVAL_MS`	no	Poll interval in milliseconds (default: `5000`)
`CELLOS_FLEET_HEARTBEAT_INTERVAL_MS`	no	Heartbeat interval in milliseconds (default: `30000`)
`CELLOS_FLEET_NODE_ID`	no	Unique node identifier (default: hostname)

The agent inherits AWS credentials from the environment (IAM role, env vars, or instance metadata) — it does not manage its own identity.

§Drain / graceful shutdown

On SIGTERM the poll loop stops accepting new work. Any in-flight cell finishes normally before the process exits. A clean drain log line is emitted so operators can distinguish graceful shutdown from a crash.

Structs§

Config: Runtime configuration resolved from environment variables at startup.
NodeIdentity: Stable identity a fleet node presents to a control plane.
PreflightChecks: The pre-flight preconditions a fleet node checks before dispatching (S29).

Enums§

CeilingEpochDecision: Outcome of the advisory pre-claim ceiling-epoch floor check (S26).
FleetProfile: Admission profile governing fail-open vs fail-closed behaviour (S29).
ProfileAdmission: Outcome of the profile-gated admission decision (S29).

Constants§

CEILING_EPOCH_STALE: Reason code emitted when the advisory floor refuses a claim (S26).
FLEET_PROFILE_ENV: Environment variable selecting the fleet admission profile (S29).
IDENTITY_KIND_KEY_POSSESSION: Identity-kind marker stamped into a key-possession attestation (S25).
MIN_CEILING_EPOCH_ENV: Environment variable carrying the advisory minimum ceiling-epoch floor (S26).
NODE_ATTESTATION_EVENT_TYPE: CloudEvent type carried by a fleet node-attestation event (S25).

Functions§

build_node_attestation: Build a signed node-attestation event for identity (S25).
ceiling_epoch_admits: Pure advisory decision: does observed_epoch clear the floor? (S26)
download_spec: Download the claimed spec to a local temp file and return its path.
finalize: Move the claimed spec to completed/ or failed/ based on exit code.
list_pending: List pending spec keys from the S3 queue prefix.
peek_pending_spec: T11 — peek a pending spec without claiming it, so we can run the placement gate before stealing the work from another pool’s runner.
process_spec: Process one spec: (T11 placement peek →) claim → download → run → finalize.
profile_admits: Fold pre-flight preconditions against the active profile (S29).
read_min_ceiling_epoch: Read the advisory ceiling-epoch floor from MIN_CEILING_EPOCH_ENV (S26).
read_spec_pool_id: Read spec.placement.poolId from a spec JSON file on disk. Returns Ok(None) when the spec has no placement block or no poolId within it. Returns Err if the file cannot be read or parsed.
run: Main poll loop with heartbeat emission and SIGTERM drain.
run_cell: Run cellos-supervisor with the downloaded spec file.
try_claim: Attempt to claim a pending spec key by moving it to the claimed prefix.
verify_node_attestation: Verify a signed node-attestation envelope offline (S25).