1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
//! Run a stream pipeline from one TOML file, with no control plane anywhere.
//!
//! The engine (`fv-streams-engine`) asks its host four questions through the
//! [`ControlPlane`](fv_streams_engine::ControlPlane) trait: what is the pipeline's topology, which topic backs a dataset, should I
//! keep running, and here is my build record. This crate answers them from a file. The same engine
//! that a platform hosts over HTTP therefore runs from a laptop, a container, or a test with the
//! same semantics: checkpoints, exactly-once output, stateful windows and joins.
//!
//! ```no_run
//! use std::sync::atomic::AtomicBool;
//! use std::sync::Arc;
//! use fv_streams::Pipeline;
//!
//! # tokio::runtime::Runtime::new().unwrap().block_on(async {
//! let stop = Arc::new(AtomicBool::new(false)); // set it to stop cleanly (Ctrl-C does)
//! Pipeline::load("pipeline.toml")?.run(stop).await;
//! # Ok::<(), String>(())
//! # });
//! ```
//!
//! A pipeline file names its datasets (Kafka topics), its transforms (the engine's own step
//! vocabulary: inline steps, `wasm`/`container` compute, `windowedAggregate`, `sessionAggregate`,
//! `streamJoin`, `topN`, `lastN`) and optional engine settings. The pipeline's `name` is its stable
//! identity: the consumer group `fv-stream-<name>`, the state topic `fv-state-<name>` and the
//! exactly-once transactional namespace all derive from it, so a restarted run restores its
//! checkpoints and fences its own zombies.
/// Every Rust code block in the README is compiled and run as a doctest.
;
use ;
pub use FileControlPlane;
pub use Pipeline;
/// `STREAM_SERVE`: the runner's observability listener (`--serve`).
pub const SERVE: Setting = Setting ;
static RUNNER: Section = Section ;
/// Every setting this runner reads, by area — the engine's, the connectors', its own — what a
/// pipeline's `[settings]` and the environment are checked against, and what `docs/settings.md`
/// lists.