# graceful-worker
Cooperative shutdown, and a retry backoff that **knows about it** — so a
fifteen-minute wait is abandoned the moment `SIGTERM` lands.
```rust
use graceful_worker::{Backoff, Shutdown};
# async fn example() {
let shutdown = Shutdown::new();
shutdown.listen_for_signals();
let watcher = shutdown.watcher();
let mut backoff = Backoff::new();
while watcher.is_running() {
match do_some_work().await {
Ok(()) => backoff.succeeded(),
Err(_) => {
backoff.failed();
// False means a stop arrived mid-wait: leave, do not retry.
if !backoff.wait(&watcher).await {
break;
}
}
}
}
# }
# async fn do_some_work() -> Result<(), ()> { Ok(()) }
```
## The problem
A container platform stops a process by sending `SIGTERM` and then waiting a
fixed grace period — often thirty seconds — before `SIGKILL`. Two things have
to be true inside that window, and they pull against each other.
**Work in flight must finish.** A worker that has taken a message off a queue
and not yet acknowledged it must not be cancelled mid-message, or the message
is lost. So shutdown here is *cooperative*: `Shutdown` asks loops to stop at
their next opportunity rather than cancelling whatever is running.
**A long retry wait must not outlast the grace period.** This is the part
other backoff crates get wrong.
## What this adds over the other backoff crates
`backoff`, `backon` and `exponential-backoff` compute a schedule and sleep
it. None of them takes a shutdown signal, so a worker that must stop promptly
has to wrap every wait in its own `select!` against whatever it uses for
cancellation — and get that right at each call site.
`Backoff::wait` takes a `Watcher` and returns `false` when a stop arrived
mid-wait:
```rust
use std::time::Duration;
use graceful_worker::{Backoff, Shutdown};
# async fn example() {
let shutdown = Shutdown::new();
let watcher = shutdown.watcher();
let mut backoff = Backoff::new();
shutdown.stop();
for _ in 0..20 { backoff.failed(); } // now at the 15-minute ceiling
// Abandoned at once, rather than in fifteen minutes.
assert!(!backoff.wait(&watcher).await);
# }
```
## On slicing, honestly
`Backoff::wait` sleeps the delay in bounded slices, and **that is not what
makes it interruptible.** Each slice is a `Watcher::sleep`, which is a
`select!` against the cancellation token and returns the moment a stop
arrives — a single unsliced sleep would be exactly as responsive.
Slicing buys nothing for shutdown latency, and version 0.1.0 and 0.1.1 of
this README claimed otherwise. It is kept as a knob (`with_slice`, or
`Duration::ZERO` to disable) because it bounds the granularity of the wait
for anyone who wants that. If you have no such need, leave it alone.
The integration with `Watcher` is the reason to use this crate.
## Configuring the schedule
The defaults — five seconds, doubling to a fifteen-minute ceiling — are a
working set, not a recommendation. What matters for a given system is the
ceiling against your platform's grace period, and only you know either.
```rust
use std::time::Duration;
use graceful_worker::Backoff;
let backoff = Backoff::new()
.with_initial_delay(Duration::from_millis(100))
.with_max_delay(Duration::from_secs(30))
.with_slice(Duration::from_secs(5))
.with_factor(3);
```
Guarded against the ways this goes wrong: an initial delay longer than the
ceiling is clamped rather than counting downwards, a factor of zero is treated
as one rather than retrying instantly forever, and a slice is never zero while
time remains — which is what stops `wait` spinning.
## Two decisions worth knowing
**Dropping a `Shutdown` does not stop anything.** A worker's shutdown must be
something someone asked for, not a consequence of where a value happened to go
out of scope. The alternative is a refactor that moves a binding and silently
turns a long-running process into one that exits immediately.
**`failed()` returns the delay that applied to *this* failure**, not the
multiplied value the next one will use — so a metric charts the actual
schedule.
## Features
- `tracing` *(default)* — a few log lines about signals and stops.
## License
MIT or Apache-2.0, at your option.