# 🔠Telemetry
> One accurate measurement is worth a thousand expert opinions.
>
> — _Grace Hopper_.
[Tracing][tracing] is a technique used in software development to understand how a system is performing and where issues might be occurring. It involves tracking the flow of requests as they move through a system, and collecting data about each step along the way. This data is then used to create a detailed picture of how the system is behaving, and can help developers identify performance bottlenecks or other issues.
To implement [tracing], developers use a technique called "[spans]". A span is a unit of work in a system, such as a function call or a network request. Each span is given a unique identifier and is linked to other [spans] that are related to it. This allows developers to follow the flow of requests through a system and understand how they are being processed.
[Tracing][tracing] is closely related to logging, which is another technique used in software development to capture information about system behavior. However, while logging provides a record of events that have occurred in a system, [tracing] provides a more detailed view of how those events are related to each other and how they are affecting system performance.
[Tracing][tracing] is an important part of the overall telemetry story, which involves collecting data about a system's behavior and using that data to improve its performance. Metrics are another important part of telemetry, and involve collecting numerical data about system performance, such as response times or error rates.
## Tools
[Prometheus](https://prometheus.io/) is a popular open-source tool for collecting and storing metrics data. It provides a powerful query language for analyzing metrics, and can be used to generate alerts when certain conditions are met. [OpenTelemetry](https://opentelemetry.io/) is another open-source project that provides a standard way of collecting telemetry data from different types of systems and applications. It provides standards and specifications on how to structure, name and format telemetry data. Today it is _the_ standard and also the one Rama follows.
Tools like [Jaeger](https://www.jaegertracing.io/) are used to visualize and analyze [tracing] data. Jaeger provides a user interface for exploring [spans] and understanding how they are related to each other. It can also be used to generate reports and identify patterns in system behavior. Overall, [tracing] is an important technique for understanding how a system is performing and can help developers identify and fix issues before they become major problems.
## Rama Telemetry
Rama re-exports [OpenTelemetry](https://opentelemetry.io/) crates under [the `rama::opentelemetry` module](https://ramaproxy.org/docs/rama/telemetry/opentelemetry/index.html),
and provides middlewares for collecting metrics on:
- the http layer: <https://ramaproxy.org/docs/rama/http/layer/opentelemetry/index.html>
- the transport layer: <https://ramaproxy.org/docs/rama/net/stream/layer/opentelemetry/index.html>
Rama ships OTLP exporters built on the Rama network stack, so you can export
telemetry without depending on `opentelemetry-otlp`.
### Rama Telemetry Example
Rama supports OTLP exporters out of the box using the Rama network stack. You
can find example source code here:
- HTTP OTLP exporter: [/examples/http_telemetry.rs](https://github.com/plabayo/rama/tree/main/examples/http_telemetry.rs)
- gRPC OTLP exporter: [/examples/grpc/src/otel_exporter.rs](https://github.com/plabayo/rama/tree/main/examples/grpc/src/otel_exporter.rs)
These examples show web services which keep track of a visitor counter as a
custom OpenTelemetry counter metric. They also make use of the Rama-provided
[`RequestMetricsLayer`](https://ramaproxy.org/docs/rama/http/layer/opentelemetry/struct.RequestMetricsLayer.html)
and
[`NetworkMetricsLayer`](https://ramaproxy.org/docs/rama/net/stream/layer/opentelemetry/struct.NetworkMetricsLayer.html)
to provide insight into traffic on both the network and application (HTTP)
layers. These metrics are exported using the OTLP exporters built into Rama.
With that example setup you can use a tool like [Grafana](https://grafana.com/) or [Prometheus](https://prometheus.io/) to make a dashboard with your own charts.
[tracing]: https://tracing.rs/tracing/
[spans]: https://tracing.rs/tracing/#spans