graceful-worker

Cooperative shutdown, and a retry backoff that sleeps in slices so a long wait never delays noticing SIGTERM.

use graceful_worker::{Backoff, Shutdown};

# async fn example() {
let shutdown = Shutdown::new();
shutdown.listen_for_signals();

let watcher = shutdown.watcher();
let mut backoff = Backoff::new();

while watcher.is_running() {
    match do_some_work().await {
        Ok(()) => backoff.succeeded(),
        Err(_) => {
            backoff.failed();
            // False means a stop arrived mid-wait: leave, do not retry.
            if !backoff.wait(&watcher).await {
                break;
            }
        }
    }
}
# }
# async fn do_some_work() -> Result<(), ()> { Ok(()) }

The problem

A container platform stops a process by sending SIGTERM and then waiting a fixed grace period — often thirty seconds — before SIGKILL. Two things have to be true inside that window, and they pull against each other.

Work in flight must finish. A worker that has taken a message off a queue and not yet acknowledged it must not be cancelled mid-message, or the message is lost. So shutdown here is cooperative: Shutdown asks loops to stop at their next opportunity rather than cancelling whatever is running.

A long retry wait must not outlast the grace period. This is the part other backoff crates get wrong.

The slicing

backoff, backon and exponential-backoff all sleep monolithically. A fifteen-minute backoff is one fifteen-minute sleep, and a SIGTERM arriving one second into it is noticed fifteen minutes later — long after the platform gave up waiting and sent SIGKILL. The process dies mid-message, which is the exact outcome cooperative shutdown existed to prevent.

Backoff::wait takes a Watcher and sleeps in bounded slices, so the wait is interruptible at every slice boundary regardless of the total delay:

use std::time::Duration;
use graceful_worker::{Backoff, Shutdown};

# async fn example() {
let shutdown = Shutdown::new();
let watcher = shutdown.watcher();
let mut backoff = Backoff::new();
shutdown.stop();

for _ in 0..20 { backoff.failed(); }   // now at the 15-minute ceiling

// Abandoned at once, rather than in fifteen minutes.
assert!(!backoff.wait(&watcher).await);
# }

The slice is the worst case for how long a stop goes unnoticed during a wait, so set it comfortably shorter than your platform's grace period.

Configuring the schedule

The defaults — five seconds, doubling to a fifteen-minute ceiling, sliced at a minute — are a working set, not a recommendation. What matters for a given system is the relationship between the slice and the grace period, and only you know either.

use std::time::Duration;
use graceful_worker::Backoff;

let backoff = Backoff::new()
    .with_initial_delay(Duration::from_millis(100))
    .with_max_delay(Duration::from_secs(30))
    .with_slice(Duration::from_secs(5))
    .with_factor(3);

Guarded against the ways this goes wrong: an initial delay longer than the ceiling is clamped rather than counting downwards, a factor of zero is treated as one rather than retrying instantly forever, and a slice is never zero while time remains — which is what stops wait spinning.

Two decisions worth knowing

Dropping a Shutdown does not stop anything. A worker's shutdown must be something someone asked for, not a consequence of where a value happened to go out of scope. The alternative is a refactor that moves a binding and silently turns a long-running process into one that exits immediately.

failed() returns the delay that applied to this failure, not the multiplied value the next one will use — so a metric charts the actual schedule.

Features

• tracing (default) — a few log lines about signals and stops.

License

MIT or Apache-2.0, at your option.