1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
//! Cooperative shutdown, and a retry backoff that sleeps in slices.
//!
//! Every long-running worker has the same skeleton: do a unit of work,
//! repeat until told to stop, and wait a bit longer each time something
//! goes wrong. This crate is that skeleton's two hard parts.
//!
//! # The problem it exists for
//!
//! A container platform stops a process by sending `SIGTERM` and then
//! waiting a fixed grace period — often thirty seconds — before `SIGKILL`.
//! Two things have to be true inside that window, and they pull against
//! each other:
//!
//! 1. **Work in flight must finish.** A worker that has taken a message off
//! a queue and not yet acknowledged it must not be cancelled mid-message,
//! or the message is lost. So shutdown is *cooperative*: [`Shutdown`]
//! asks loops to stop at their next opportunity rather than cancelling
//! whatever is running.
//!
//! 2. **A long retry wait must not outlast the grace period.** This is the
//! part other backoff crates get wrong. `backoff`, `backon` and
//! `exponential-backoff` all sleep monolithically, so a fifteen-minute
//! backoff means `SIGTERM` is noticed up to fifteen minutes later — long
//! after `SIGKILL`. [`Backoff::wait`] sleeps in bounded slices against a
//! [`Watcher`], so the wait is interruptible at every slice boundary no
//! matter how long the total delay is.
//!
//! # The loop
//!
//! ```
//! # #[tokio::main(flavor = "current_thread")]
//! # async fn main() {
//! use graceful_worker::{Backoff, Shutdown};
//!
//! let shutdown = Shutdown::new();
//! shutdown.listen_for_signals();
//!
//! let watcher = shutdown.watcher();
//! let mut backoff = Backoff::new();
//!
//! # shutdown.stop();
//! while watcher.is_running() {
//! match do_some_work().await {
//! Ok(()) => backoff.succeeded(),
//! Err(_) => {
//! backoff.failed();
//! // False means a stop arrived mid-wait: leave, do not retry.
//! if !backoff.wait(&watcher).await {
//! break;
//! }
//! }
//! }
//! }
//! # async fn do_some_work() -> Result<(), ()> { Ok(()) }
//! # }
//! ```
//!
//! # Two decisions worth knowing
//!
//! **Dropping a [`Shutdown`] does not stop anything.** A worker's shutdown
//! must be something someone asked for, not a consequence of where a value
//! happened to go out of scope.
//!
//! **The schedule is configurable and the defaults are not a
//! recommendation.** Five seconds, doubling to fifteen minutes, sliced at a
//! minute. What matters for a given system is the relationship between the
//! slice and the platform's grace period, and only the caller knows either.
//!
//! # Features
//!
//! - `tracing` *(default)* — a few log lines about signals and stops.
pub use Backoff;
pub use ;