graceful-worker

Cooperative shutdown, and a retry backoff that knows about it — so a fifteen-minute wait is abandoned the moment SIGTERM lands.

use graceful_worker::{Backoff, Shutdown};

# async fn example() {
let shutdown = Shutdown::new();
shutdown.listen_for_signals();

let watcher = shutdown.watcher();
let mut backoff = Backoff::new();

while watcher.is_running() {
    match do_some_work().await {
        Ok(()) => backoff.succeeded(),
        Err(_) => {
            backoff.failed();
            // False means a stop arrived mid-wait: leave, do not retry.
            if !backoff.wait(&watcher).await {
                break;
            }
        }
    }
}
# }
# async fn do_some_work() -> Result<(), ()> { Ok(()) }

The problem

A container platform stops a process by sending SIGTERM and then waiting a fixed grace period — often thirty seconds — before SIGKILL. Two things have to be true inside that window, and they pull against each other.

Work in flight must finish. A worker that has taken a message off a queue and not yet acknowledged it must not be cancelled mid-message, or the message is lost. So shutdown here is cooperative: Shutdown asks loops to stop at their next opportunity rather than cancelling whatever is running.

A long retry wait must not outlast the grace period. This is the part other backoff crates get wrong.

What this adds over the other backoff crates

backoff, backon and exponential-backoff compute a schedule and sleep it. None of them takes a shutdown signal, so a worker that must stop promptly has to wrap every wait in its own select! against whatever it uses for cancellation — and get that right at each call site.

Backoff::wait takes a Watcher and returns false when a stop arrived mid-wait:

use std::time::Duration;
use graceful_worker::{Backoff, Shutdown};

# async fn example() {
let shutdown = Shutdown::new();
let watcher = shutdown.watcher();
let mut backoff = Backoff::new();
shutdown.stop();

for _ in 0..20 { backoff.failed(); }   // now at the 15-minute ceiling

// Abandoned at once, rather than in fifteen minutes.
assert!(!backoff.wait(&watcher).await);
# }

On slicing, honestly

Backoff::wait sleeps the delay in bounded slices, and that is not what makes it interruptible. Each slice is a Watcher::sleep, which is a select! against the cancellation token and returns the moment a stop arrives — a single unsliced sleep would be exactly as responsive.

Slicing buys nothing for shutdown latency, and version 0.1.0 and 0.1.1 of this README claimed otherwise. It is kept as a knob (with_slice, or Duration::ZERO to disable) because it bounds the granularity of the wait for anyone who wants that. If you have no such need, leave it alone.

The integration with Watcher is the reason to use this crate.

Configuring the schedule

The defaults — five seconds, doubling to a fifteen-minute ceiling — are a working set, not a recommendation. What matters for a given system is the ceiling against your platform's grace period, and only you know either.

use std::time::Duration;
use graceful_worker::Backoff;

let backoff = Backoff::new()
    .with_initial_delay(Duration::from_millis(100))
    .with_max_delay(Duration::from_secs(30))
    .with_slice(Duration::from_secs(5))
    .with_factor(3);

Guarded against the ways this goes wrong: an initial delay longer than the ceiling is clamped rather than counting downwards, a factor of zero is treated as one rather than retrying instantly forever, and a slice is never zero while time remains — which is what stops wait spinning.

Two decisions worth knowing

Dropping a Shutdown does not stop anything. A worker's shutdown must be something someone asked for, not a consequence of where a value happened to go out of scope. The alternative is a refactor that moves a binding and silently turns a long-running process into one that exits immediately.

failed() returns the delay that applied to this failure, not the multiplied value the next one will use — so a metric charts the actual schedule.

Features

• tracing (default) — a few log lines about signals and stops.

License

MIT or Apache-2.0, at your option.