1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
//! A library for building Arpx runtimes. This library provides an interface for constructing Arpx
//! runtimes with code or via a profile.
//!
//! # Anatomy of a runtime
//!
//! The basic anatomy of a typical runtime is as follows:
//!
//! ```text
//! +---------+
//! | Runtime |
//! +---------+
//! |
//! | +------------+
//! +->| Job: "dev" |
//! +------------+
//! |
//! | +------+
//! +->| Task |
//! +------+
//! |
//! | +---------------------+
//! +--->| Process: "database" |
//! | +---------------------+
//! | |
//! | +-------+ +-----------+
//! | | Fail? |------>| Self heal |
//! | +-------+ +-----------+
//! |
//! | +----------------+
//! +--->| Process: "api" |
//! +----------------+
//! |
//! +-------+ +-----------+
//! | Fail? |------>| Self heal |
//! +-------+ +-----------+
//! ```
//!
//! A runtime contains one or more jobs, each of which contains one or more tasks, each of which
//! contains one or more processes. Multiple processes in a single task will run concurrently, and
//! any `onsucceed` or `onfail` actions will be performed on the same thread as the process which
//! spawned them.
//!
//! Multiple tasks in a single job will execute in sequence, in the order in which they're defined on
//! the job object. The same is true for multiple jobs in a single runtime.
//!
//! In the above diagram, the runtime contains one job (`dev`). `dev` contains one task with
//! multiple processes, `database` and `api`. These processes will run concurrently. Both of these
//! processes declare an `onfail` action which performs some sort of self-healing procedure and,
//! presumably, respawns the failed process.
//!
//! So, runtimes can run processes concurrently or in sequence, as well as respond to success and
//! failure states upon process exit.
//!
//! # Define a runtime using this library
//!
//! A runtime similar to the one diagrammed above can be built with code:
//!
//! ```
//! use arpx::{Job, LogMonitor, Process, Runtime, Task};
//! use std::collections::HashMap;
//!
//! let processes = vec![
//! Process::new("database".to_string())
//! .command("run.sh".to_string())
//! .cwd("/path/to/project/database/".to_string())
//! .onsucceed(Some("db_recover".to_string())),
//! Process::new("api".to_string())
//! .command("run.sh".to_string())
//! .cwd("/path/to/project/api/".to_string())
//! .onsucceed(Some("api_recover".to_string())),
//! ];
//!
//! let mut process_map = processes
//! .clone()
//! .into_iter()
//! .map(|process| (process.name.clone(), process))
//! .collect::<HashMap<String, Process>>();
//!
//! process_map.insert(
//! "db_recover".to_string(),
//! Process::new("db_recover".to_string())
//! .command("self-heal.sh".to_string())
//! .cwd("/path/to/project/database/".to_string())
//! .onsucceed(Some("database".to_string()))
//! .onfail(Some("arpx_exit_error".to_string()))
//! );
//!
//! process_map.insert(
//! "api_recover".to_string(),
//! Process::new("api_recover".to_string())
//! .command("self-heal.sh".to_string())
//! .cwd("/path/to/project/api/".to_string())
//! .log_monitors(vec!["db_permissions_error".to_string()])
//! .onsucceed(Some("api".to_string()))
//! .onfail(Some("arpx_exit_error".to_string())),
//! );
//!
//! let mut log_monitor_map = HashMap::new();
//!
//! log_monitor_map.insert(
//! "db_permissions_error".to_string(),
//! LogMonitor::new("db_permissions_error".to_string())
//! .buffer_size(1)
//! .test("echo \"$ARPX_BUFFER\" | grep -q \"Access denied for user\"".to_string())
//! .ontrigger("arpx_exit_error".to_string())
//! );
//!
//! let jobs = vec![Job::new(
//! "dev".to_string(),
//! vec![Task::new(processes)],
//! )];
//!
//! Runtime::new()
//! .jobs(jobs)
//! .process_map(process_map)
//! .log_monitor_map(log_monitor_map)
//! .run();
//! ```
//!
//! ## About log monitors
//!
//! Note in the above example that the `api_recover` process is provided with a list of
//! `log_monitors`. The runtime isn't limited to handling exit codes; it can be configured to
//! respond to runtime errors as well.
//!
//! Log monitors allow for string matching against a rolling buffer of a given process's output.
//! For example, the `db_permissions_error` log monitor is applied to the `database` process in job
//! `dev`. The `db_permisions_error` log monitor will keep a rolling buffer of size 1 (the 1 most
//! recent line of the process it's watching) and run its `test` script on every push to the
//! buffer. If the script returns with a `0` exit status, the `ontrigger` action will run. In the
//! case of `db_permissions_error`, a successful `test` will exit the entire runtime.
//!
//! # Define a runtime using a profile
//!
//! This runtime can also be defined in a profile:
//!
//! ```yaml
//! jobs:
//! dev: |
//! [
//! database : db_recover; @db_permissions_error
//! api : api_recover;
//! ]
//!
//! processes:
//! database:
//! command: run.sh
//! cwd: /path/to/project/database/
//! onfail: db_recover
//! db_recover:
//! command: self-heal.sh
//! cwd: /path/to/project/database/
//! onsucceed: database
//! onfail: arpx_exit_error
//! api:
//! command: run.sh
//! cwd: /path/to/project/api/
//! onfail: api_recover
//! api_recover:
//! command: self-heal.sh
//! cwd: /path/to/project/api/
//! onsucceed: api
//! onfail: arpx_exit_error
//!
//! log_monitors:
//! db_permissions_error:
//! buffer_size: 1
//! test: 'echo "$ARPX_BUFFER" | grep -q "Access denied for user"'
//! ontrigger: arpx_exit_error
//! ```
//!
//! A runtime object can be built from this profile by loading the profile using
//! [`Runtime.from_profile()`].
//!
//! [`Runtime.from_profile`]: Runtime#method.from_profile
//!
//! ## The advantage of using a profile
//!
//! The advantage of using a profile is expressiveness. The `process_map` and `log_monitor_map` are
//! defined nicely using YAML maps and the details of how each job orchestrates the available
//! processes and log monitors into an effective runtime are expressed succinctly using the
//! dedicated [arpx_job scripting language][arpx_job docs]:
//!
//! ```text
//! [
//! database : db_recover; @db_permissions_error
//! api : api_recover;
//! ]
//! ```
//!
//! ### Some syntax notes
//!
//! Because these processes are enclosed in square brackets (`[`, `]`), the runtime knows to group
//! them together in the same task. This means that `database` and `api` will be executed
//! concurrently, as they should.
//!
//! `database` is followed by `: db_recover`. This is an `onfail` declaration. The runtime will
//! parse this and know that it needs to run `db_recover` if the `database` process exits with a
//! non-zero code.
//!
//! If `database` were followed by `? db_recover` instead, `db_recover` would run `onsucceed`
//! instead of `onfail`. If the declaration were `database ? db_recover : db_recover`, then
//! `db_recover` would run no matter what. Note that this is [ternary
//! operator](https://en.wikipedia.org/wiki/%3F:) syntax.
//!
//! Also, the `@db_permissions_error` declaration on the `database` task tells the runtime to apply
//! the `db_permissions_error` log monitor to the `database` process.
//!
//! **Related:** [arpx_job documentation][arpx_job docs]
//!
//! [arpx_job docs]: https://github.com/jaredgorski/arpx/tree/main/docs/writing_a_profile.md#arpx_job-scripting-language
pub use Logs;
pub use ;