graceful_worker/

lib.rs

//! Cooperative shutdown, and a retry backoff that sleeps in slices.
//!
//! Every long-running worker has the same skeleton: do a unit of work,
//! repeat until told to stop, and wait a bit longer each time something
//! goes wrong. This crate is that skeleton's two hard parts.
//!
//! # The problem it exists for
//!
//! A container platform stops a process by sending `SIGTERM` and then
//! waiting a fixed grace period — often thirty seconds — before `SIGKILL`.
//! Two things have to be true inside that window, and they pull against
//! each other:
//!
//! 1. **Work in flight must finish.** A worker that has taken a message off
//!    a queue and not yet acknowledged it must not be cancelled mid-message,
//!    or the message is lost. So shutdown is *cooperative*: [`Shutdown`]
//!    asks loops to stop at their next opportunity rather than cancelling
//!    whatever is running.
//!
//! 2. **A long retry wait must not outlast the grace period.**
//!    `backoff`, `backon` and `exponential-backoff` compute a schedule and
//!    sleep it; none takes a shutdown signal, so a worker has to wrap every
//!    wait in its own `select!` and get that right at each call site.
//!    [`Backoff::wait`] takes a [`Watcher`] and returns `false` when a stop
//!    arrived mid-wait, so a fifteen-minute backoff is abandoned the moment
//!    `SIGTERM` lands rather than fifteen minutes later.
//!
//! # The loop
//!
//! ```
//! # #[tokio::main(flavor = "current_thread")]
//! # async fn main() {
//! use graceful_worker::{Backoff, Shutdown};
//!
//! let shutdown = Shutdown::new();
//! shutdown.listen_for_signals();
//!
//! let watcher = shutdown.watcher();
//! let mut backoff = Backoff::new();
//!
//! # shutdown.stop();
//! while watcher.is_running() {
//!     match do_some_work().await {
//!         Ok(()) => backoff.succeeded(),
//!         Err(_) => {
//!             backoff.failed();
//!             // False means a stop arrived mid-wait: leave, do not retry.
//!             if !backoff.wait(&watcher).await {
//!                 break;
//!             }
//!         }
//!     }
//! }
//! # async fn do_some_work() -> Result<(), ()> { Ok(()) }
//! # }
//! ```
//!
//! # Two decisions worth knowing
//!
//! **Dropping a [`Shutdown`] does not stop anything.** A worker's shutdown
//! must be something someone asked for, not a consequence of where a value
//! happened to go out of scope.
//!
//! **The schedule is configurable and the defaults are not a
//! recommendation.** Five seconds, doubling to fifteen minutes. What
//! matters for a given system is the ceiling against the platform's grace
//! period, and only the caller knows either.
//!
//! **Slicing is a knob, not the mechanism.** [`Backoff::wait`] sleeps in
//! slices, but that is not what makes it interruptible — each slice is a
//! `select!` against the cancellation token, so an unsliced sleep would be
//! just as responsive. See the [`backoff`] module documentation.
//!
//! # Features
//!
//! - `tracing` *(default)* — a few log lines about signals and stops.

#![cfg_attr(test, allow(clippy::unwrap_used, clippy::expect_used, clippy::panic))]

pub mod backoff;
pub mod shutdown;

pub use backoff::Backoff;
pub use shutdown::{Shutdown, Watcher};