metrique 0.1.25

Library for generating wide event metrics
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

# Sampling

If your service's TPS is too high for full metric emission, sampling lets you
reduce volume while preserving visibility into rare events. See also the
["My TPS is too high"]
section in the cookbook.

## Overview

High-volume services may want to trade lower accuracy for lower CPU time spent on metric emission. Offloading metrics to
CloudWatch can become bottlenecked if the CloudWatch Agent or other log forwarder isn't able to keep up with the rate of written metric entries.

It is common to tee the metric into 2 destinations:

 1. A highly-compressed "log of record" that contains all entries and is eventually persisted to S3 or other long-term storage.
 1. An uncompressed, but sampled, metrics log that is published to CloudWatch.

The sampling can be done naively at some [fixed fraction], but at low rates can
cause low-frequency events to be missed. This includes service errors or validation errors, especially when the service is
designed to have an availability much higher than the chosen sample rate. Instead, we recommend the use of the
[congressional sampler]. It uses a fixed metric emission target rate and
gives lower-frequency events a higher sampling rate to boost their accuracy.

The example below uses the congressional sampler keyed by the request operation and the status code to
ensure lower-frequency APIs and status codes have enough samples.

When using EMF, you need to call [`with_sampling`] before calling a sampler, for example:

```rust,no_run
use metrique::unit_of_work::metrics;
use metrique::emf::Emf;
use metrique::writer::{AttachGlobalEntrySinkExt, FormatExt, GlobalEntrySink};
use metrique::writer::sample::SampledFormatExt;
use metrique::writer::stream::tee;
use metrique::ServiceMetrics;
use tracing_appender::rolling::{RollingFileAppender, Rotation};

# let service_log_dir = "./service_log";
# let metrics_log_dir = "./metrics_log";

#[metrics(value(string))]
enum Operation {
    CountDucks,
    // ...
}

#[metrics(rename_all="PascalCase")]
struct RequestMetrics {
    #[metrics(sample_group)]
    operation: Operation,
    #[metrics(sample_group)]
    status_code: &'static str,
    number_of_ducks: u32,
    exception: Option<String>,
}

let _join_service_metrics = ServiceMetrics::attach_to_stream(
    tee(
        // non-uploaded, archived log of record
        Emf::all_validations("MyNS".to_string(), /* dimensions */ vec![vec![], vec!["Operation".to_string()]])
            .output_to_makewriter(RollingFileAppender::new(
                Rotation::MINUTELY,
                service_log_dir,
                "service_log.log",
            )),
        // sampled log, will be uploaded to CloudWatch
        Emf::all_validations("MyNS".to_string(), /* dimensions */ vec![vec![], vec!["Operation".to_string()]])
            .with_sampling()
            .sample_by_congress_at_fixed_entries_per_second(100)
            .output_to_makewriter(RollingFileAppender::new(
                Rotation::MINUTELY,
                metrics_log_dir,
                "metric_log.log",
            )),
    )
);

let metric = RequestMetrics {
    operation: Operation::CountDucks,
    status_code: "OK",
    number_of_ducks: 2,
    exception: None,
}.append_on_drop(ServiceMetrics::sink());

// _join_service_metrics drop (e.g. during service shutdown) blocks until the queue is drained
```

[`with_sampling`]: https://docs.rs/metrique/latest/metrique/emf/struct.Emf.html#method.with_sampling
["My TPS is too high"]: https://docs.rs/metrique/latest/metrique/_guide/cookbook/#my-tps-is-too-high
[congressional sampler]: https://docs.rs/metrique/latest/metrique/writer/sample/struct.CongressSample.html
[fixed fraction]: https://docs.rs/metrique/latest/metrique/writer/sample/struct.FixedFractionSample.html