# graceful-worker
Cooperative shutdown, and a retry backoff that **sleeps in slices** so a long
wait never delays noticing `SIGTERM`.
```rust
use graceful_worker::{Backoff, Shutdown};
# async fn example() {
let shutdown = Shutdown::new();
shutdown.listen_for_signals();
let watcher = shutdown.watcher();
let mut backoff = Backoff::new();
while watcher.is_running() {
match do_some_work().await {
Ok(()) => backoff.succeeded(),
Err(_) => {
backoff.failed();
// False means a stop arrived mid-wait: leave, do not retry.
if !backoff.wait(&watcher).await {
break;
}
}
}
}
# }
# async fn do_some_work() -> Result<(), ()> { Ok(()) }
```
## The problem
A container platform stops a process by sending `SIGTERM` and then waiting a
fixed grace period — often thirty seconds — before `SIGKILL`. Two things have
to be true inside that window, and they pull against each other.
**Work in flight must finish.** A worker that has taken a message off a queue
and not yet acknowledged it must not be cancelled mid-message, or the message
is lost. So shutdown here is *cooperative*: `Shutdown` asks loops to stop at
their next opportunity rather than cancelling whatever is running.
**A long retry wait must not outlast the grace period.** This is the part
other backoff crates get wrong.
## The slicing
`backoff`, `backon` and `exponential-backoff` all sleep monolithically. A
fifteen-minute backoff is one fifteen-minute `sleep`, and a `SIGTERM` arriving
one second into it is noticed fifteen minutes later — long after the platform
gave up waiting and sent `SIGKILL`. The process dies mid-message, which is the
exact outcome cooperative shutdown existed to prevent.
`Backoff::wait` takes a `Watcher` and sleeps in bounded slices, so the wait is
interruptible at every slice boundary regardless of the total delay:
```rust
use std::time::Duration;
use graceful_worker::{Backoff, Shutdown};
# async fn example() {
let shutdown = Shutdown::new();
let watcher = shutdown.watcher();
let mut backoff = Backoff::new();
shutdown.stop();
for _ in 0..20 { backoff.failed(); } // now at the 15-minute ceiling
// Abandoned at once, rather than in fifteen minutes.
assert!(!backoff.wait(&watcher).await);
# }
```
The slice is the worst case for how long a stop goes unnoticed during a wait,
so set it comfortably shorter than your platform's grace period.
## Configuring the schedule
The defaults — five seconds, doubling to a fifteen-minute ceiling, sliced at a
minute — are a working set, not a recommendation. What matters for a given
system is the relationship between the slice and the grace period, and only
you know either.
```rust
use std::time::Duration;
use graceful_worker::Backoff;
let backoff = Backoff::new()
.with_initial_delay(Duration::from_millis(100))
.with_max_delay(Duration::from_secs(30))
.with_slice(Duration::from_secs(5))
.with_factor(3);
```
Guarded against the ways this goes wrong: an initial delay longer than the
ceiling is clamped rather than counting downwards, a factor of zero is treated
as one rather than retrying instantly forever, and a slice is never zero while
time remains — which is what stops `wait` spinning.
## Two decisions worth knowing
**Dropping a `Shutdown` does not stop anything.** A worker's shutdown must be
something someone asked for, not a consequence of where a value happened to go
out of scope. The alternative is a refactor that moves a binding and silently
turns a long-running process into one that exits immediately.
**`failed()` returns the delay that applied to *this* failure**, not the
multiplied value the next one will use — so a metric charts the actual
schedule.
## Features
- `tracing` *(default)* — a few log lines about signals and stops.
## License
MIT or Apache-2.0, at your option.