Crate metrique_writer

Expand description

This crate contains infrastructure to make writers that emit wide events (structured metric records).

If you want to write a library or application that generates wide events, the API in this crate is lower-level than you’ll want, and it’s normally much easier to do that via the #[metrics] macro in the metrique crate, and therefore, the main Getting Started documentation is in that crate - it is recommended to read that, even if you end up using just metrique-writer.

§Metric Entries

This crate is centered around an Entry trait. Implementations of the Entry trait define a metric that can be emitted in various formats

§Writing Entries

The easiest way to make metric entries is by using the metrique crate, see the docs for the metrique crate for that.

It is also possible to make Entryes without the metrique crate, by using #[derive(Entry)]

use metrique_writer::{Entry, unit::{AsBytes, AsSeconds}};

use std::time::Duration;

#[derive(Entry)]
pub struct MyInnerEntry {
    inner_value: u32
}

#[derive(Entry)]
pub struct MyEntry {
    property: String,
    value: u32,
    // emitted as 0 or 1
    flag: bool,
    // emitted as milliseconds, as per `impl Value for Duration`
    time: Duration,
    // but you can use AsSeconds etc. to change it to other units
    // to fill it in, use `my_duration.into()`
    time_seconds: AsSeconds<Duration>,
    // you can also use the unit markers to give units to other values
    value_bytes: AsBytes<u32>,
    // if None, not emitted; if Some, emitted as the inner value
    optional_value: Option<AsBytes<u32>>,
    // you can flatten entries
    #[entry(flatten)]
    inner_entry: MyInnerEntry,
}

It is also possible to implement Entry manually (see the docs for Entry).

§Emitting Metrics

This library currently only comes with metrique-writer-format-emf, which formats to Amazon CloudWatch Embedded Metric Format (EMF), but more formatters might be added in the future.

You can also implement a custom format using the Format trait. If you do, you can optionally implement a custom EntrySink if you need flush functionality beyond writing bytes to an arbitrary I/O destination.

Entries are sent to an EntrySink in order to be written to a destination.

You can either thread the EntrySink manually in your code, or register a global entry sink by using the sink::global_entry_sink macro.

The EntrySink generally has these 3 components:

A Format that is used to format the metrics to be emitted.
Some output file, which turns the Format to an EntryIoStream
Some queueing policy (sink::BackgroundQueue or sink::FlushImmediately).

One example way of setting up metrics would be to do the following:

use metrique_writer::{
    Entry, GlobalEntrySink,
    format::FormatExt as _,
    sink::{AttachGlobalEntrySinkExt, global_entry_sink},
    unit::AsCount,
};
use metrique_writer_format_emf::Emf;
use tracing_appender::rolling::{RollingFileAppender, Rotation};

global_entry_sink! {
    /// A special metrics sink for my application
    MyEntrySink
}


#[derive(Entry, Default)]
struct MyMetrics {
    field: AsCount<usize>,
}

#[derive(Entry)]
struct Globals {
    region: String,
}

let globals = Globals {
    // Generally, this is usually sourced from CLI args or the environment
    region: "us-east-1".to_string(),
};

let join = MyEntrySink::attach_to_stream(Emf::all_validations("MyApp".into(), 
    vec![vec![], vec!["region".into()]] /* emit using the [] and ["region"] dimension sets */)
    // All entries will contain `region: us-east-1` as a property
    .merge_globals(globals)
    .output_to_makewriter(
        RollingFileAppender::new(Rotation::HOURLY, log_dir, "prefix.log")
    )
);

// you might want to detach the queue - otherwise, dropping the `BackgroundQueueJoinHandle`
// will shut the BackgroundQueue down and then wait for it to be flushed.
// join.forget();

let mut metric = MyEntrySink::append_on_drop_default::<MyMetrics>();
*metric.field += 1;
// metric appends to sink as scope ends and variable drops

You can also do it without a global entry sink, if you’ll rather not use a global variable:

use metrique_writer::{
    Entry,
    EntrySink,
    format::FormatExt as _,
    sink::BackgroundQueue,
    unit::AsCount,
};
use metrique_writer_format_emf::Emf;
use tracing_appender::rolling::{RollingFileAppender, Rotation};


#[derive(Entry, Default)]
struct MyMetrics {
    field: AsCount<usize>,
}

#[derive(Entry)]
struct Globals {
    region: String,
}


// this will create a BackgroundQueue that works only with the type MyMetrics. This
// is slightly faster, since there is no boxing

let globals = Globals {
    // Generally, this is usually sourced from CLI args or the environment
    region: "us-east-1".to_string(),
};

let (queue, join) = BackgroundQueue::<MyMetrics>::new(Emf::all_validations("MyApp".into(), 
    vec![vec![], vec!["region".into()]] /* emit using the [] and ["region"] dimension sets */)
    // All entries will contain `region: us-east-1` as a property
    .merge_globals(globals)
    .output_to_makewriter(
        RollingFileAppender::new(Rotation::HOURLY, log_dir, "prefix.log")
    )
);

// you might want to detach the queue - otherwise, dropping the `BackgroundQueueJoinHandle`
// will shut the BackgroundQueue down and then wait for it to be flushed.
// join.forget();

let mut metric = queue.append_on_drop_default();
*metric.field += 1;
// metric appends to sink as scope ends and variable drops

Or without a global entry sink, but still with a queue that accepts multiple metric types:

use metrique_writer::{
    Entry,
    BoxEntry,
    AnyEntrySink,
    format::FormatExt as _,
    sink::BackgroundQueueBuilder,
    unit::AsCount,
};
use metrique_writer_format_emf::Emf;
use tracing_appender::rolling::{RollingFileAppender, Rotation};


#[derive(Entry, Default)]
struct MyMetrics {
    field: AsCount<usize>,
}

#[derive(Entry)]
struct Globals {
    region: String,
}


// this will create a BackgroundQueue that works only with the type MyMetrics. This
// is slightly faster, since there is no boxing

let globals = Globals {
    // Generally, this is usually sourced from CLI args or the environment
    region: "us-east-1".to_string(),
};

let (queue, join) = BackgroundQueueBuilder::new().build_boxed(
    Emf::all_validations("MyApp".into(), 
        vec![vec![], vec!["region".into()]] /* emit using the [] and ["region"] dimension sets */)
        // All entries will contain `region: us-east-1` as a property
        .merge_globals(globals)
        .output_to_makewriter(
            RollingFileAppender::new(Rotation::HOURLY, log_dir, "prefix.log")
        )
);

// you might want to detach the queue - otherwise, dropping the `BackgroundQueueJoinHandle`
// will shut the BackgroundQueue down and then wait for it to be flushed.
// join.forget();

let mut metric = MyMetrics::default();
*metric.field += 1;
queue.append_any(metric);

Re-exports§

pub use crate::sink::AttachGlobalEntrySinkExt;
pub use format::FormatExt;
pub use stream::EntryIoStreamExt;

Modules§

entry: Contains various utilities for Entry
format: Contains various utilities for Format
rate_limit: Cross-thread rate limiting for noisy log call sites. See rate_limited!.
sample: Contains samplers that can be used to sample metrics to avoid performance impact when the metric rate is high.
sink: Contains various utilities for working with EntrySink
stream: Contains various utilities for working with EntryIoStream
test_utiltest-util: TestEntry provides a way to directly introspect the result of writing out fields with Entry
unit: Contains utilities for attaching Units (such as percents, kilobytes, or seconds) to metrics. Conversion between different units is handled by Convert.
value: Contains various utilities for working with Value.

Macros§

rate_limited: rate_limited!(duration, expr) will cause expr to only be evaluated at most once every duration across all threads.

Structs§

BoxEntry: A heap-allocated Entry wrapper that uses dynamic dispatch.
BoxEntrySink: A type-erased EntrySink, that can sink a BoxEntry (which can contain an arbitrary Entry).
Distribution: A flag indicating that this metric represents a distribution (histogram).
MetricFlags: Contains a set of options that describe the implementation of a metric
ShutdownFn: A function to be called during shutdown when the AttachHandle is dropped.
ValidationError: An error type that describes why an crate::Entry isn’t valid. This can be because it violated general contracts (e.g. writing multiple values with the same name) or because it violated a format-specific contract (e.g. using a reserved property name).
ValidationErrorBuilder: Builder to record validation failures over time and bundle them into a single ValidationError Note that if no validation failures are added to the builder, ValidationErrorBuilder::build() will return None, which is useful to track if a side-effect produced any errors.

Enums§

IoStreamError: The error cases for a EntryIoStream::next call.
Observation: The numeric value of a observation to include in a metric value.
Unit: Represent all metric value units allowed by CloudWatch.

Traits§

AnyEntrySink: Provides a more generic interface than EntrySink but may come at the cost of dynamic dispatch and heap allocation to store the in-memory buffer.
AttachGlobalEntrySink: A GlobalEntrySink that can do nothing until it is attached to an output stream or sink.
Convert: When implemented, signifies that values with the unit Self can be converted to the unit U by multiplying by Convert::RATIO.
Entry: The core trait to be implemented by application data structures holding metric values.
EntryConfig: Trait for format-specific Entry configuration, formats will downcast this to the specific config
EntryIoStream: Writes a stream of entries to an output IO sink.
EntrySink: Stores entries in an in-memory buffer until they can be written to the destination.
EntryWriter: Provided by a format for each atomic entry that will be written to the metric destination.
GlobalEntrySink: A global version of crate::EntrySink that can be referred to by any thread or component.
MetricValue: A Value type that promises to write a metric with unit MetricValue::Unit.
Value: A metric value that may be associated with a name in a crate::EntryWriter::value() call.
ValueWriter: Provided by a format for each call to crate::EntryWriter::value().

Derive Macros§

Entry: Derive Entry for a struct or enum.