Struct CircuitStats 

pub struct CircuitStats { /* private fields */ }

Circuit breaker statistics for tracking state transitions.

Maintains a rolling window of successes and failures to determine when the circuit should open, close, or enter half-open state.

Implementations

impl CircuitStats

pub fn new() -> Self

Create new circuit stats in the closed state.

pub fn consecutive_command_failures(&self) -> u32

Consecutive command-class failures observed since the last command-class success (or circuit close).

pub fn reopen_streak(&self) -> u32

Open transitions since the circuit last closed (0 when it has not opened since recovery). Exposed for diagnostics.

pub fn state(&self) -> CircuitState

Get the current circuit state.

pub fn opened_at(&self) -> Option<u64>

Get the timestamp when the circuit opened (if open).

pub fn last_state_change(&self) -> u64

Get the timestamp of the last state change.

pub fn consecutive_failures(&self) -> u32

Get the consecutive failure count.

pub fn error_rate(&self) -> f64

Get the error rate in the current window (0.0-1.0).

Computed over the bounded recent-results window (the last CIRCUIT_HISTORY_SIZE health checks) rather than the unbounded lifetime window_successes/window_failures counters. A long-lived daemon polling distant workers accumulates occasional transient failures over days; with a lifetime ratio a currently-healthy worker could stay gated (circuit kept open, depressed health score) until the daemon was restarted. A recency window lets the rate self-heal as fresh successes age out old failures.

pub fn record_success(&mut self)

Record a successful operation.

In closed state: resets consecutive failures, increments window successes. In half-open state: increments consecutive successes.

pub fn record_failure(&mut self)

Record a failed operation.

Increments consecutive failures and window failures. In half-open state, resets consecutive successes.

pub fn should_open(&self, config: &CircuitBreakerConfig) -> bool

Check if the circuit should open based on config thresholds.

Returns true if:

• Consecutive failures exceed threshold, OR
• Error rate exceeds threshold (with minimum sample size)

pub fn should_half_open(&self, config: &CircuitBreakerConfig) -> bool

Check if the circuit should transition to half-open.

Returns true if the circuit is open and the cooldown period has elapsed.

pub fn should_close(&self, config: &CircuitBreakerConfig) -> bool

Check if the circuit should close.

Returns true if in half-open state, consecutive successes meet or exceed the success threshold, AND no command-class failure is outstanding. A worker whose last real command timed out is not recovered no matter how many liveness probes it answers; it stays HalfOpen (degraded, probe-limited) until a command-class success clears the evidence.

pub fn can_probe(&self, config: &CircuitBreakerConfig) -> bool

Check if a probe request can be made in half-open state.

Returns true if active probes are below the maximum allowed.

pub fn start_probe(&mut self, config: &CircuitBreakerConfig) -> bool

Start a probe request in half-open state.

Returns true if the probe was started, false if already at max probes.

pub fn open(&mut self)

Transition the circuit to open state.

pub fn half_open(&mut self)

Transition the circuit to half-open state.

pub fn close(&mut self)

Transition the circuit to closed state.

pub fn reset_window(&mut self)

Reset the rolling window counters.

Called periodically to ensure the window reflects recent activity. Also clears the recent-results history that error_rate() is computed from, so a reset truly zeroes the observed error rate.

pub fn recent_results(&self) -> &[bool]

Get recent health check results for history visualization.

Returns a slice of recent results (true=success, false=failure), with the most recent result at the end.

pub fn recovery_remaining_secs( &self, config: &CircuitBreakerConfig, ) -> Option<u64>

Calculate seconds remaining until circuit auto-transitions to half-open.

Returns None if circuit is not open or cooldown has already elapsed.

pub fn apply_health_outcome( &mut self, healthy: bool, transient: bool, config: &CircuitBreakerConfig, ) -> CircuitState

Drive the circuit through a single health-check outcome and return the resulting CircuitState.

This is the single, self-contained transition engine shared by every circuit consumer (the daemon health monitor via WorkerState::record_health_check, and the diagnostic WorkerHealth panel via WorkerHealth::observe). Keeping the transition logic in ONE place is what makes WorkerState.circuit (which selection reads) and the diagnostic WorkerHealth.circuit agree.

Behaviour:

• On a healthy outcome: if the circuit is Open and its cooldown has elapsed it first transitions Open → HalfOpen before crediting the success, so recovery needs exactly success_threshold clean probes (crediting a success against an Open circuit would be lost). Then the success is recorded and the circuit closes once the threshold is met.
• On an unhealthy outcome: the failure is recorded, the circuit opens if the closed-state open condition is met, and a HalfOpen probe failure reopens the circuit — UNLESS the failure was transient (a retryable transport blip that survived the in-check retries). A transient half-open failure neither reopens nor resets the cooldown, so a single network hiccup during recovery cannot bounce a worker back to Open.
• After either branch, an Open circuit past its cooldown is nudged to HalfOpen so probing resumes on the next poll even for a still-failing worker.

pub fn apply_command_outcome( &mut self, success: bool, config: &CircuitBreakerConfig, ) -> CircuitState

Drive the circuit through a single command-class outcome and return the resulting CircuitState.

Command-class evidence comes from an operation of the same shape as dispatched work: a remote binary that must run to completion (the telemetry poll, or a health probe whose SSH session connected but whose command never finished). It is the signal the plain liveness probe cannot provide — a host under fork exhaustion or swap thrash keeps answering echo over a warm connection while every real command hangs, and on that evidence alone the breaker would happily re-admit it.

Behaviour:

• Success: clears the outstanding command-failure count and then counts as an ordinary healthy outcome (may advance Open -> HalfOpen -> Closed exactly like a liveness success).
• Failure: increments the command-failure count and records a plain failure. The circuit opens when either the ordinary open condition or command_failure_threshold is met. In HalfOpen a command failure ALWAYS reopens — there is no transient exemption, because “connected but the command did not complete” is precisely the condition the half-open trial exists to detect. While any command failure is outstanding, Self::should_close refuses to close on liveness successes alone.

Trait Implementations

impl Clone for CircuitStats

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CircuitStats

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CircuitStats

fn default() -> Self

Returns the “default value” for a type.