metrics_runtime/

lib.rs

//! High-speed metrics collection library.
//!
//! `metrics-runtime` provides a generalized metrics collection library targeted at users who want
//! to log metrics at high volume and high speed.
//!
//! # Design
//!
//! The library follows a pattern of "senders" and a "receiver."
//!
//! Callers create a [`Receiver`], which acts as a registry for all metrics that flow through it.
//! It allows creating new sinks as well as controllers, both necessary to push in and pull out
//! metrics from the system.  It also manages background resources necessary for the registry to
//! operate.
//!
//! Once a [`Receiver`] is created, callers can either create a [`Sink`] for sending metrics, or a
//! [`Controller`] for getting metrics out.
//!
//! A [`Sink`] can be cheaply cloned, and offers convenience methods for getting the current time
//! as well as getting direct handles to a given metric.  This allows users to either work with the
//! fuller API exposed by [`Sink`] or to take a compositional approach and embed fields that
//! represent each particular metric to be sent.
//!
//! A [`Controller`] provides both a synchronous and asynchronous snapshotting interface, which is
//! [`metrics-core`][metrics_core] compatible for exporting.  This allows flexibility in
//! integration amongst traditional single-threaded or hand-rolled multi-threaded applications and
//! the emerging asynchronous Rust ecosystem.
//!
//! # Performance
//!
//! Users can expect to be able to send tens of millions of samples per second, with ingest
//! latencies at roughly 65-70ns at p50, and 250ns at p99.  Depending on the workload -- counters
//! vs histograms -- latencies may be even lower, as counters and gauges are markedly faster to
//! update than histograms.  Concurrent updates of the same metric will also cause natural
//! contention and lower the throughput/increase the latency of ingestion.
//!
//! # Metrics
//!
//! Counters, gauges, and histograms are supported, and follow the definitions outlined in
//! [`metrics-core`][metrics_core].
//!
//! Here's a simple example of creating a receiver and working with a sink:
//!
//! ```rust
//! # extern crate metrics_runtime;
//! use metrics_runtime::Receiver;
//! use std::{thread, time::Duration};
//! let receiver = Receiver::builder().build().expect("failed to create receiver");
//! let mut sink = receiver.sink();
//!
//! // We can update a counter.  Counters are monotonic, unsigned integers that start at 0 and
//! // increase over time.
//! sink.increment_counter("widgets", 5);
//!
//! // We can update a gauge.  Gauges are signed, and hold on to the last value they were updated
//! // to, so you need to track the overall value on your own.
//! sink.update_gauge("red_balloons", 99);
//!
//! // We can update a timing histogram.  For timing, we're using the built-in `Sink::now` method
//! // which utilizes a high-speed internal clock.  This method returns the time in nanoseconds, so
//! // we get great resolution, but giving the time in nanoseconds isn't required!  If you want to
//! // send it in another unit, that's fine, but just pay attention to that fact when viewing and
//! // using those metrics once exported.  We also support passing `Instant` values -- both `start`
//! // and `end` need to be the same type, though! -- and we'll take the nanosecond output of that.
//! let start = sink.now();
//! thread::sleep(Duration::from_millis(10));
//! let end = sink.now();
//! sink.record_timing("db.queries.select_products_ns", start, end);
//!
//! // Finally, we can update a value histogram.  Technically speaking, value histograms aren't
//! // fundamentally different from timing histograms.  If you use a timing histogram, we do the
//! // math for you of getting the time difference, but other than that, identical under the hood.
//! let row_count = 46;
//! sink.record_value("db.queries.select_products_num_rows", row_count);
//! ```
//!
//! # Scopes
//!
//! Metrics can be scoped, not unlike loggers, at the [`Sink`] level.  This allows sinks to easily
//! nest themselves without callers ever needing to care about where they're located.
//!
//! This feature is a simpler approach to tagging: while not as semantically rich, it provides the
//! level of detail necessary to distinguish a single metric between multiple callsites.
//!
//! For example, after getting a [`Sink`] from the [`Receiver`], we can easily nest ourselves under
//! the root scope and then send some metrics:
//!
//! ```rust
//! # extern crate metrics_runtime;
//! # use metrics_runtime::Receiver;
//! # let receiver = Receiver::builder().build().expect("failed to create receiver");
//! // This sink has no scope aka the root scope.  The metric will just end up as "widgets".
//! let mut root_sink = receiver.sink();
//! root_sink.increment_counter("widgets", 42);
//!
//! // This sink is under the "secret" scope.  Since we derived ourselves from the root scope,
//! // we're not nested under anything, but our metric name will end up being "secret.widgets".
//! let mut scoped_sink = root_sink.scoped("secret");
//! scoped_sink.increment_counter("widgets", 42);
//!
//! // This sink is under the "supersecret" scope, but we're also nested!  The metric name for this
//! // sample will end up being "secret.supersecret.widget".
//! let mut scoped_sink_two = scoped_sink.scoped("supersecret");
//! scoped_sink_two.increment_counter("widgets", 42);
//!
//! // Sinks retain their scope even when cloned, so the metric name will be the same as above.
//! let mut cloned_sink = scoped_sink_two.clone();
//! cloned_sink.increment_counter("widgets", 42);
//!
//! // This sink will be nested two levels deeper than its parent by using a slightly different
//! // input scope: scope can be a single string, or multiple strings, which is interpreted as
//! // nesting N levels deep.
//! //
//! // This metric name will end up being "super.secret.ultra.special.widgets".
//! let mut scoped_sink_three = scoped_sink.scoped(&["super", "secret", "ultra", "special"]);
//! scoped_sink_two.increment_counter("widgets", 42);
//! ```
//!
//! # Labels
//!
//! On top of scope support, metrics can also have labels. If scopes are for organizing metrics in
//! a hierarchy, then labels are for differentiating the same metric being emitted from multiple
//! sources.
//!
//! This is most easily demonstrated with an example:
//!
//! ```rust
//! # extern crate metrics_runtime;
//! # fn run_query(_: &str) -> u64 { 42 }
//! # use metrics_runtime::Receiver;
//! # let receiver = Receiver::builder().build().expect("failed to create receiver");
//! # let mut sink = receiver.sink();
//! // We might have a function that interacts with a database and returns the number of rows it
//! // touched in doing so.
//! fn process_query(query: &str) -> u64 {
//!     run_query(query)
//! }
//!
//! // We might call this function multiple times, but hitting different tables.
//! let rows_a = process_query("UPDATE posts SET public = 1 WHERE public = 0");
//! let rows_b = process_query("UPDATE comments SET public = 1 WHERE public = 0");
//!
//! // Now, we want to track a metric that shows how many rows are updated overall, so the metric
//! // name should be the same no matter which table we update, but we'd also like to be able to
//! // differentiate by table, too!
//! sink.record_value_with_labels("db.rows_updated", rows_a, &[("table", "posts")]);
//! sink.record_value_with_labels("db.rows_updated", rows_b, &[("table", "comments")]);
//!
//! // If you want to send a specific set of labels with every metric from this sink, you can also
//! // add default labels.  This action is additive, so you can call it multiple times to build up
//! // the set of labels sent with metrics, and labels are inherited when creating a scoped sink or
//! // cloning an existing sink, which allows label usage to either supplement scopes or to
//! // potentially replace them entirely.
//! sink.add_default_labels(&[("database", "primary")]);
//! # fn main() {}
//! ```
//!
//! As shown in the example, labels allow a user to submit values to the underlying metric name,
//! while also differentiating between unique situations, whatever the facet that the user decides
//! to utilize.
//!
//! Naturally, these methods can be slightly cumbersome and visually detracting, in which case
//! you can utilize the metric handles -- [`Counter`](crate::data::Counter),
//! [`Gauge`](crate::data::Gauge), and [`Histogram`](crate::data::Histogram) -- and create them
//! with labels ahead of time.
//!
//! These handles are bound to the given metric type, as well as the name, labels, and scope of the
//! sink.  Thus, there is no overhead of looking up the metric as with the `record_*` methods, and
//! the values can be updated directly, and with less overhead, resulting in faster method calls.
//!
//! ```rust
//! # extern crate metrics_runtime;
//! # use metrics_runtime::Receiver;
//! # use std::time::Instant;
//! # let receiver = Receiver::builder().build().expect("failed to create receiver");
//! # let mut sink = receiver.sink();
//! // Let's create a counter.
//! let egg_count = sink.counter("eggs");
//!
//! // I want a baker's dozen of eggs!
//! egg_count.increment();
//! egg_count.record(12);
//!
//! // This updates the same metric as above!  We have so many eggs now!
//! sink.increment_counter("eggs", 12);
//!
//! // Gauges and histograms don't have any extra helper methods, just `record`:
//! let gauge = sink.gauge("population");
//! gauge.record(8_000_000_000);
//!
//! let histogram = sink.histogram("distribution");
//!
//! // You can record a histogram value directly:
//! histogram.record_value(42);
//!
//! // Or handily pass it two [`Delta`]-compatible values, and have it calculate the delta for you:
//! let start = Instant::now();
//! let end = Instant::now();
//! histogram.record_timing(start, end);
//!
//! // Each of these methods also has a labels-aware companion:
//! let labeled_counter = sink.counter_with_labels("egg_count", &[("type", "large_brown")]);
//! let labeled_gauge = sink.gauge_with_labels("population", &[("country", "austria")]);
//! let labeled_histogram = sink.histogram_with_labels("distribution", &[("type", "performance")]);
//! # fn main() {}
//! ```
//!
//! # Proxies
//!
//! Sometimes, you may have a need to pull in "external" metrics: values related to your
//! application that your application itself doesn't generate, such as system-level metrics.
//!
//! [`Sink`] allows you to register a "proxy metric", which gives the ability to return metrics
//! on-demand when a snapshot is being taken.  Users provide a closure that is run every time a
//! snapshot is being taken, which can return multiple metrics, which are then added to overall
//! list of metrics being held by `metrics-runtime` itself.
//!
//! If metrics are relatively expensive to calculate -- say, accessing the /proc filesytem on Linux
//! -- then this can be a great alternative to polling them yourself and having to update them
//! normally on some sort of schedule.
//!
//! ```rust
//! # extern crate metrics_runtime;
//! # extern crate metrics_core;
//! # use metrics_core::Key;
//! # use metrics_runtime::{Receiver, Measurement};
//! # use std::time::Instant;
//! # let receiver = Receiver::builder().build().expect("failed to create receiver");
//! # let mut sink = receiver.sink();
//! // A proxy is now registered under the name "load_stats", which is prepended to all the metrics
//! // generated by the closure i.e. "load_stats.avg_1min".  These metrics are also still scoped
//! // normally based on the [`Sink`].
//! sink.proxy("load_stat", || {
//!     let mut values = Vec::new();
//!     values.push((Key::from_name("avg_1min"), Measurement::Gauge(19)));
//!     values.push((Key::from_name("avg_5min"), Measurement::Gauge(12)));
//!     values.push((Key::from_name("avg_10min"), Measurement::Gauge(10)));
//!     values
//! });
//! # fn main() { }
//! ```
//!
//! # Snapshots
//!
//! Naturally, we need a way to get the metrics out of the system, which is where snapshots come
//! into play.  By utilizing a [`Controller`], we can take a snapshot of the current metrics in the
//! registry, and then output them to any desired system/interface by utilizing
//! [`Observer`](metrics_core::Observer).  A number of pre-baked observers (which only concern
//! themselves with formatting the data) and exporters (which take the formatted data and either
//! serve it up, such as exposing an HTTP endpoint, or write it somewhere, like stdout) are
//! available, some of which are exposed by this crate.
//!
//! Let's take an example of writing out our metrics in a yaml-like format, writing them via
//! `log!`:
//! ```rust
//! # extern crate metrics_runtime;
//! use metrics_runtime::{
//!     Receiver, observers::YamlBuilder, exporters::LogExporter,
//! };
//! use log::Level;
//! use std::{thread, time::Duration};
//! let receiver = Receiver::builder().build().expect("failed to create receiver");
//! let mut sink = receiver.sink();
//!
//! // We can update a counter.  Counters are monotonic, unsigned integers that start at 0 and
//! // increase over time.
//! // Take some measurements, similar to what we had in other examples:
//! sink.increment_counter("widgets", 5);
//! sink.update_gauge("red_balloons", 99);
//!
//! let start = sink.now();
//! thread::sleep(Duration::from_millis(10));
//! let end = sink.now();
//! sink.record_timing("db.queries.select_products_ns", start, end);
//! sink.record_timing("db.gizmo_query", start, end);
//!
//! let num_rows = 46;
//! sink.record_value("db.queries.select_products_num_rows", num_rows);
//!
//! // Now create our exporter/observer configuration, and wire it up.
//! let exporter = LogExporter::new(
//!     receiver.controller(),
//!     YamlBuilder::new(),
//!     Level::Info,
//!     Duration::from_secs(5),
//! );
//!
//! // This exporter will now run every 5 seconds, taking a snapshot, rendering it, and writing it
//! // via `log!` at the informational level. This particular exporter is running directly on the
//! // current thread, and not on a background thread.
//! //
//! // exporter.run();
//! ```
//! Most exporters have the ability to run on the current thread or to be converted into a future
//! which can be spawned on any Tokio-compatible runtime.
//!
//! # Facade
//!
//! `metrics-runtime` is `metrics` compatible, and can be installed as the global metrics facade:
//! ```
//! # #[macro_use] extern crate metrics;
//! extern crate metrics_runtime;
//! use metrics_runtime::Receiver;
//!
//! Receiver::builder()
//!     .build()
//!     .expect("failed to create receiver")
//!     .install();
//!
//! counter!("items_processed", 42);
//! ```
//!
//! [metrics_core]: https://docs.rs/metrics-core
//! [`Observer`]: https://docs.rs/metrics-core/0.3.1/metrics_core/trait.Observer.html
#![deny(missing_docs)]
#![warn(unused_extern_crates)]
mod builder;
mod common;
mod config;
mod control;
pub mod data;
mod helper;
mod receiver;
mod registry;
mod sink;

#[cfg(any(feature = "metrics-exporter-log", feature = "metrics-exporter-http"))]
pub mod exporters;

#[cfg(any(
    feature = "metrics-observer-yaml",
    feature = "metrics-observer-json",
    feature = "metrics-observer-prometheus"
))]
pub mod observers;

pub use self::{
    builder::{Builder, BuilderError},
    common::{Delta, Measurement, Scope},
    control::Controller,
    receiver::Receiver,
    sink::{AsScoped, Sink, SinkError},
};