etherbird 0.4.3

Stable async clients with connection recovery and session restoration
Documentation

etherbird

Keep the client. Replace the connection.

etherbird keeps async clients usable through connection failures. Define how to create, connect, and set up a resource once; the library replaces failed resources, restores their configuration and session, and waits for setup to finish before running application calls. Your application keeps using the same client.

It works with network connections, device handles, and other resources that need more than reconnection to become usable again. Your adapter supplies the protocol logic, such as authentication, subscriptions, or a device handshake.

Useful features

  • A stable, typed client: managed_client! generates ordinary async methods around your resource, keeping acquisition and recovery out of application calls.
  • Restored configuration: named values and setters apply to existing resources and every replacement; stored values remain readable during an outage.
  • Recovery while idle: an optional disconnect watcher detects connection loss even when no operation is running. Old failure reports cannot retire a replacement.
  • Bounded recovery: configurable lifecycle deadlines and capped exponential connection backoff prevent callers from managing their own reconnect loops.
  • Adapter-controlled failure policies: suspend lifecycle attempts that need corrected configuration, or return a healthy operation error without reconnecting.
  • Checked configuration: validate lifecycle backoff and pool bounds before starting work, with field-specific errors from fallible constructors.
  • Single-attempt call timeouts: bound readiness, queueing and operation duration without opting into replay. Shutdown takes precedence over timeout expiry.
  • Bounded FIFO admission: use BoundedFifoQueue with a queue factory to reject excess waiting calls without writing a custom queue.
  • Adapter cancellation helpers: RetireOnDrop clears an incomplete exchange's resource slot and signals connection loss. The adapter explicitly confirms valid replies, including healthy device rejections.
  • Exclusive-scope interruption: DisconnectOnDrop signals connection loss before releasing adapter ownership when setup or a multi-exchange scope is interrupted.
  • Generation-bound adapter state: opaque Generation tokens prevent handles from matching another connection, client or pool slot. Adapters check readiness.
  • Explicit operation retries: calls run once by default. Safe-to-repeat calls can opt into an attempt limit, overall deadline, and error predicate.
  • Observable state and orderly shutdown: connection indicators, state notifications, lifecycle callbacks, and tracing help you monitor resources. Awaited shutdown completes teardown even if another shutdown caller is cancelled.
  • Optional pooling: cap resource counts, grow on demand, retire idle resources, and reserve exclusive leases. Schedule operations with FIFO, priority, or a custom queue that can reject excess work. Pooled shutdown cancels and drains tracked work.

Start with direct supervision for a client that supports concurrent calls. Enable pooling when resources need exclusive access, capacity limits, or queued scheduling.

etherbird = "0.4.3"
# With pooling:
# etherbird = { version = "0.4.3", features = ["pool"] }

The API is experimental. See the managed client guide for construction and ownership, and the feature guide for lifecycle, cancellation, shutdown, and recovery guarantees.

Read the lifecycle policy guide for failure, timeout, validation and shutdown contracts, and the changelog for release history and migration notes.

Performance

In the local MQTT admission benchmarks, direct supervision performs in roughly the same range as equivalent manual coordination. Pooling adds measurable cost for tiny, high-frequency operations, especially under saturated multithreaded workloads; it also provides exclusive access, scheduling, and tracked operation shutdown. Results depend on workload and runtime.

Read the MQTT performance analysis and pool admission and allocation analysis for methodology, measurements, and raw data. MQTT timings measure local publish admission, not broker acknowledgement or message delivery.

Concrete examples

Example What it shows
MQTT application comparison The same telemetry, status, and request code with manual coordination, direct supervision, or pooling. Restores subscriptions and checks readiness, cancellation, and repeated broker outages.
Modbus and WebSocket recovery · scenario guide Bounded device connections, queued register reads, opt-in retries, and restored WebSocket subscriptions after socket failures.
Serial instrument · adapter guide Reopens a serial port, reapplies device gain, and repeats the handshake before admitting calls.
Typed client and lifecycle A small, self-contained introduction to lifecycle hooks, generated methods, direct and pooled clients, and configuration replay.
Priority and overload Urgent work overtakes telemetry, equal priorities retain FIFO order, and a bounded custom queue rejects excess requests.
Direct MQTT session Uses Supervisor directly with a stable receiver, subscription acknowledgements, and restart recovery.

Run the application examples from a checkout:

# MQTT comparison with local fixtures; no broker required:
cargo run --example mqtt_with_supervisor -- --demo
cargo run --example mqtt_without_etherbird -- --demo
cargo run --features pool --example mqtt_with_etherbird -- --demo

# Modbus and WebSocket fixtures; no external service required:
cargo run --features pool --example live_recovery

# Serial demo with Unix pseudo-terminals; no hardware required:
cargo run --features pool --example serial_device -- --demo

MQTT examples also support --broker-demo for real Mosquitto outages through Docker. See the examples index for all commands and platform requirements, and the MQTT comparison analysis for the coordination code each approach requires.

Generic usage

These snippets run inside an async function on a Tokio runtime. They assume a Hooks adapter implementing Lifecycle, whose resource exposes async fn read(&self) -> Result<u16, std::io::Error>. Implement create, connect, and disconnect; use setup for authentication, subscriptions, or a handshake. Optional hooks add cleanup, destruction, disconnect watching, live connection status, and error classification. The typed client example contains a complete adapter.

1. Supervise one resource

execute waits for a ready resource and runs the operation once. By default, an operation error requests recovery and returns the error to its caller. An adapter can classify a healthy error with OperationFailurePolicy::Retain.

use etherbird::{Config, Supervisor};

let supervisor = Supervisor::new(Hooks, Config::default());
let result = supervisor.execute(|resource| async move { resource.read().await }).await;
supervisor.stop().await;
let sample = result?;

2. Give the application ordinary client methods

Declare forwarding signatures once. The underlying resource must support concurrent calls for this direct client.

use etherbird::{Config, Supervisor};

etherbird::managed_client! {
    struct Client for Hooks {
        async fn read() -> u16;
    }
}

let client = Client::new(Supervisor::new(Hooks, Config::default()));
let result = client.read().await;
client.managed.stop().await;
let sample = result?;

3. Add bounded, exclusive access

With the pool feature, the same Client declaration can use up to four resources. Each generated method reserves one resource for its operation.

use etherbird::{Config, Pool, PoolConfig, Supervisor};

let pool = Pool::new(
    || Supervisor::new(Hooks, Config::default()),
    PoolConfig { min_size: 1, max_size: 4, ..PoolConfig::default() },
);
let client = Client::from_pool(pool);
let result = client.read().await;
client.managed.stop().await;
let sample = result?;

4. Combine recovery, configuration, scheduling, deadlines and retries

This builds on the same Client and Hooks. The resource additionally provides short synchronous set_gain(u16) and gain() -> u16 methods. Its adapter's setup restores the session and watch_disconnect reports idle connection loss.

use etherbird::{Config, Pool, PoolConfig, RetryPolicy, Supervisor};
use std::{io, num::NonZeroUsize, sync::Arc, time::Duration};

let pool = Pool::new(
    || Supervisor::new(Hooks, Config {
        resource_name: "instrument".into(),
        create_timeout: Some(Duration::from_secs(5)),
        created_timeout: Some(Duration::from_secs(5)),
        connect_timeout: Duration::from_secs(5),
        setup_timeout: Duration::from_secs(5),
        cleanup_timeout: Duration::from_secs(2),
        disconnect_timeout: Duration::from_secs(2),
        destroy_timeout: Some(Duration::from_secs(2)),
        retry_delay: Duration::from_millis(100),
        max_retry_delay: Duration::from_secs(5),
    }),
    PoolConfig {
        min_size: 1,
        max_size: 4,
        idle_timeout: Duration::from_secs(30),
        priority_queue: true,
    },
);

// Install configuration and callbacks before lazy startup.
pool.set_value("gain", 2_u16, |resource, gain| resource.set_gain(*gain));
pool.set_on_resource_created(Arc::new(|_resource| Box::pin(async {
    tracing::info!("created instrument resource");
    Ok(())
})));
let changes = pool.subscribe(); // A watch receiver for state/membership changes.
let client = Client::from_pool(pool.clone());
assert_eq!(client.managed.value::<u16>("gain"), Some(2));

let result = async {
    client.managed.connected_with_timeout(Duration::from_secs(5)).await?;
    let first = client.read().await?; // Ordinary calls still run once.
    let gain = client.managed.attribute(|resource| resource.gain());

    // Bound queueing, readiness and execution of one attempt; no replay.
    let bounded = pool.execute_with_timeout(
        Duration::from_secs(2),
        |resource| async move { resource.read().await },
    ).await?;

    // Lower numbers have higher priority. Retry this read only, within one deadline.
    let sample = pool.execute_with_priority_and_retry(
        0,
        RetryPolicy {
            max_attempts: NonZeroUsize::new(3).unwrap(),
            timeout: Duration::from_secs(10),
        },
        |resource| async move { resource.read().await },
        |error| matches!(error.kind(),
            io::ErrorKind::ConnectionReset | io::ErrorKind::UnexpectedEof),
    ).await?;

    // Reserve one resource across multiple calls, including any recovery.
    let lease = pool.borrow().await?;
    let reserved = lease.execute(|resource| async move { resource.read().await }).await?;
    lease.release();
    Ok::<_, etherbird::Error<io::Error>>((first, bounded, sample, reserved, gain))
}.await;

// Cancel and drain managed work, then tear down every resource, even on error.
client.managed.stop().await;
let readings = result?;

Use BoundedFifoQueue for bounded waiting admission, or a custom queue for another rejection/scheduling policy. FifoQueue and PriorityQueue are unbounded. State watchers can await changes.changed() and inspect pool.state(). Configure a tracing subscriber to collect lifecycle diagnostics.

5. Protect adapter handles and multi-exchange scopes

The combined application example calls an adapter that owns connection safety. Inside that adapter, create a private Generation token for each new session and bind remote handles with generation.bind(value). Validate the token and live connection state before using a handle. Hold DisconnectOnDrop::new(ownership_guard, &connected) across setup or a complete multi-exchange operation; call complete() only when releasing ownership is safe. Interruption signals loss before unlocking, even between exchanges. Keep RetireOnDrop around each actual exchange through reply validation.

The runnable adapter example demonstrates successful same-generation use, local rejection by a replacement session, and retirement on a deadline between exchanges. It runs with default features and with pooling:

cargo run --locked --example adapter_scopes
cargo run --locked --features pool --example adapter_scopes

See adapter contracts for token lifetime, readiness, ownership and teardown responsibilities.

Retry only operations that are safe to repeat: a disconnected write may already have taken effect remotely. Completion retains the underlying client's meaning; protocol or application acknowledgements belong in the adapter or application. Resources must be Send + Sync; use interior mutability where their APIs need mutable access. Direct calls run in their callers, while pooled calls are tracked and drained by the library during shutdown.

Documentation

Guide Purpose
Managed clients Choose direct supervision or pooled ownership and construct a typed client
Capabilities Current APIs and guarantees
Lifecycle policies Failure, timeout, cancellation, ownership and migration contracts
Examples MQTT, Modbus, WebSocket and serial adapter usage
Live recovery Reproduce socket fault/recovery checks and interpret their evidence
MQTT comparison Compare manual, direct and pooled coordination
Performance reports and admission measurements Reproduction methods, retained results and limitations
Open work Current handoff, next investigation and acceptance criteria
Changelog and release evidence Completed changes, publication and validation records

API reference · Current handoff and engineering tasks · CI · Real broker recovery checks

Licensed under the Apache License 2.0.