Crate graceful_worker 

Cooperative shutdown, and a retry backoff that sleeps in slices.

Every long-running worker has the same skeleton: do a unit of work, repeat until told to stop, and wait a bit longer each time something goes wrong. This crate is that skeleton’s two hard parts.

The problem it exists for

A container platform stops a process by sending SIGTERM and then waiting a fixed grace period — often thirty seconds — before SIGKILL. Two things have to be true inside that window, and they pull against each other:

1. Work in flight must finish. A worker that has taken a message off a queue and not yet acknowledged it must not be cancelled mid-message, or the message is lost. So shutdown is cooperative: Shutdown asks loops to stop at their next opportunity rather than cancelling whatever is running.

2. A long retry wait must not outlast the grace period. backoff, backon and exponential-backoff compute a schedule and sleep it; none takes a shutdown signal, so a worker has to wrap every wait in its own select! and get that right at each call site. Backoff::wait takes a Watcher and returns false when a stop arrived mid-wait, so a fifteen-minute backoff is abandoned the moment SIGTERM lands rather than fifteen minutes later.

The loop

use graceful_worker::{Backoff, Shutdown};

let shutdown = Shutdown::new();
shutdown.listen_for_signals();

let watcher = shutdown.watcher();
let mut backoff = Backoff::new();

while watcher.is_running() {
    match do_some_work().await {
        Ok(()) => backoff.succeeded(),
        Err(_) => {
            backoff.failed();
            // False means a stop arrived mid-wait: leave, do not retry.
            if !backoff.wait(&watcher).await {
                break;
            }
        }
    }
}

Two decisions worth knowing

Dropping a Shutdown does not stop anything. A worker’s shutdown must be something someone asked for, not a consequence of where a value happened to go out of scope.

The schedule is configurable and the defaults are not a recommendation. Five seconds, doubling to fifteen minutes. What matters for a given system is the ceiling against the platform’s grace period, and only the caller knows either.

Slicing is a knob, not the mechanism. Backoff::wait sleeps in slices, but that is not what makes it interruptible — each slice is a select! against the cancellation token, so an unsliced sleep would be just as responsive. See the backoff module documentation.

Features

• tracing (default) — a few log lines about signals and stops.

Re-exports

pub use backoff::Backoff;

pub use shutdown::Shutdown;

pub use shutdown::Watcher;

Modules

backoff

How long a failing operation waits before trying again — and how a stop cuts the wait short.

shutdown

Noticing SIGTERM, and stopping without losing work in progress.