Skip to main content

pipeline_viz/
lib.rs

1//! Live item-level visibility for Rust data pipelines.
2//!
3//! `tracing` shows spans, `tokio-console` shows tasks, Prometheus shows
4//! counters. None of them answer the question that actually stalls a debugging
5//! session: *where is block 2049102 right now, and why has it not moved in
6//! forty seconds?*
7//!
8//! `pipeline-viz` answers exactly that. Declare your stages, wrap each work item
9//! in a guard, and give a reason whenever an item is parked.
10//!
11//! ```no_run
12//! use pipeline_viz::{NodeKind, PipelineTracker};
13//!
14//! # async fn example() -> Result<(), Box<dyn std::error::Error>> {
15//! let tracker = PipelineTracker::builder().bind_port(9999).start_background()?;
16//! tracker.register_node("committer", NodeKind::Sink, ["indexer"]);
17//!
18//! let mut job = tracker.job("committer").id(2_049_102).job_type("Block").start();
19//! job.hold("Waiting for finality (2/12 confirmations)");
20//! // ... wait for confirmations ...
21//! job.update_reason("Writing to PostgreSQL");
22//! // ... write ...
23//! job.complete();
24//! # Ok(())
25//! # }
26//! ```
27//!
28//! # Enabling it
29//!
30//! The crate does nothing until the `viz` feature is on, and it is **off by
31//! default**. That is deliberate — nothing should be able to ship a listening
32//! port to production by accident.
33//!
34//! ```toml
35//! [dependencies]
36//! pipeline-viz = { version = "0.1", features = ["viz"] }
37//! ```
38//!
39//! With the feature off, every call above compiles to an empty inlined body and
40//! none of the implementation's dependencies are built.
41//!
42//! # The dashboard
43//!
44//! The tracker serves it on the port you bound. The whole UI is compiled into
45//! your binary — no static directory to deploy, no Node.js on the machine that
46//! runs it, nothing to configure.
47//!
48//! # Where it is safe to run
49//!
50//! The dashboard binds `127.0.0.1` and nothing else, and it is
51//! **unauthenticated**: anything that can reach the port can read every item
52//! id, hold reason and metadata in the pipeline. There is no option to bind a
53//! public address, because there is no safe way to offer one.
54//!
55//! That makes it a tool for local development, a staging box, and production
56//! diagnosis reached over an SSH tunnel:
57//!
58//! ```text
59//! ssh -N -L 9999:127.0.0.1:9999 you@your-host
60//! ```
61//!
62//! # What it does not guarantee
63//!
64//! Instrumentation never applies backpressure to the pipeline it is watching.
65//! When the event channel is full the event is dropped and counted, so under a
66//! burst the dashboard can fall behind the truth:
67//!
68//! * A nonzero [`PipelineTracker::dropped_events`] means the displayed state
69//!   **may be stale**. An enter, a completion or an abandonment can have been
70//!   among the losses, so an item may show at a node it has already left.
71//! * Nothing reconciles that afterwards. Reconnecting a browser repeats the
72//!   collector's current state; it does not repair it.
73//! * The mitigations are to raise [`TrackerBuilder::channel_capacity`], or to
74//!   restart the tracker for a clean slate.
75//!
76//! Abandoned items are retained for diagnosis up to
77//! [`TrackerBuilder::max_retained_abandoned`] records, oldest evicted first.
78//! That bounds the record count, not the bytes they carry and not active or
79//! held work.
80//!
81//! # Lifetime
82//!
83//! The dashboard belongs to the tracker. Keep a [`PipelineTracker`] alive for
84//! as long as you want it; when the last clone drops, the collector, the
85//! process sampler, the server and every open WebSocket stop and the port is
86//! released. Nothing is flushed on the way out.
87//!
88//! # Attribute macros
89//!
90//! Enable the `macros` feature for [`track_node`] and [`track_job`], which turn
91//! the calls above into two annotations:
92//!
93//! ```toml
94//! [dependencies]
95//! pipeline-viz = { version = "0.1", features = ["viz", "macros"] }
96//! ```
97//!
98//! ```
99//! # #[cfg(feature = "macros")] mod example {
100//! use pipeline_viz::{track_job, track_node};
101//!
102//! #[track_node(id = "committer", kind = Sink, name = "Database Committer", inputs = ["indexer"])]
103//! #[track_job(node = "committer", id = number, job_type = "Block", meta(tx_count = 142))]
104//! async fn commit(number: u64) -> Result<(), &'static str> {
105//!     // ... write to PostgreSQL ...
106//!     Ok(())
107//! }
108//! # }
109//! ```
110//!
111//! `track_node` registers the stage on the annotated function's first call.
112//! `track_job` opens a [`JobGuard`] for the duration of the body and completes
113//! it when the body returns — including an early `return` or a `?` that yielded
114//! an `Err`. A panic drops the guard instead, which is what marks an item
115//! abandoned.
116//!
117//! Both read the tracker from [`install`], because an annotated function has
118//! nowhere to receive a handle. They expand to the same runtime calls shown
119//! above and hold no state of their own, so the two surfaces cannot drift
120//! apart. With nothing installed — or with `viz` off — the generated calls do
121//! nothing.
122
123#![forbid(unsafe_code)]
124#![warn(missing_debug_implementations)]
125#![warn(missing_docs)]
126
127pub mod model;
128
129mod global;
130
131#[cfg(feature = "viz")]
132mod assets;
133#[cfg(feature = "viz")]
134mod cancel;
135#[cfg(feature = "viz")]
136mod collector;
137#[cfg(feature = "viz")]
138mod event;
139#[cfg(feature = "viz")]
140mod process;
141#[cfg(feature = "viz")]
142mod runtime;
143#[cfg(feature = "viz")]
144mod server;
145#[cfg(feature = "viz")]
146mod tracker;
147
148#[cfg(not(feature = "viz"))]
149mod noop;
150
151#[cfg(feature = "viz")]
152pub use tracker::{Error, JobBuilder, JobGuard, PipelineTracker, TrackerBuilder};
153
154#[cfg(not(feature = "viz"))]
155pub use noop::{Error, JobBuilder, JobGuard, PipelineTracker, TrackerBuilder};
156
157pub use global::{global, install, InstallError};
158
159pub use model::{
160    JobId, JobPhase, JobState, NodeCounters, NodeId, NodeKind, NodeState, Patch, ProcessStats,
161    ServerMessage, Snapshot,
162};
163
164/// Registers the annotated function as a pipeline node.
165///
166/// See the [module-level macro documentation](crate#attribute-macros).
167#[cfg(feature = "macros")]
168pub use pipeline_viz_macros::track_node;
169
170/// Tracks one work item for the duration of the annotated function.
171///
172/// See the [module-level macro documentation](crate#attribute-macros).
173#[cfg(feature = "macros")]
174pub use pipeline_viz_macros::track_job;