Expand description
otlp2records - Transform OTLP telemetry to Arrow RecordBatches
This crate provides synchronous, WASM-compatible transformation of OpenTelemetry Protocol (OTLP) data (logs, traces, metrics) into Arrow RecordBatches.
§Design Principles
- No I/O: Core never touches network or filesystem
- No async: Pure synchronous transforms
- WASM-first: All dependencies compile to wasm32
- Arrow-native: RecordBatch is the canonical output format
§High-level API
The simplest way to use this crate is with the high-level transform functions:
ⓘ
use otlp2records::{transform_logs, transform_traces, transform_metrics, InputFormat};
let batch = transform_logs(bytes, InputFormat::Protobuf)?;
let batch = transform_traces(bytes, InputFormat::Json)?;
let batches = transform_metrics(bytes, InputFormat::Protobuf)?;Modules§
- fixtures
- Synthetic OTLP protobuf fixture encoders.
- otap
- Stateful native OTAP input.
- proto_
output - Protobuf output helpers for compatibility endpoints.
Structs§
- Json
Metric Batches - Result of transforming OTLP metrics to JSON values.
- Metric
Batches - Result of transforming OTLP metrics to Arrow RecordBatches.
- Partitioned
Batch - A RecordBatch with partition metadata for storage routing.
- Partitioned
Metrics - Partitioned metrics result, with each metric type grouped by service.
- Service
Grouped Batches - Multiple batches grouped by service name.
- Skipped
Metrics - Tracks metrics that were skipped during transformation.
- Transform
Counter Value - A counted transform event.
- Transform
Phase Timing - A measured transform phase.
Enums§
- Decode
Error - Errors that can occur during OTLP decoding.
- Error
- Errors that can occur during OTLP to RecordBatch transformation
- Input
Format - Input format for OTLP decoding.
- Transform
Counter - Counted transform event.
- Transform
Phase - Timed transform phase.
- Transform
Signal - Telemetry signal being transformed.
Traits§
- Transform
Observer - Optional sink for transform timing observations.
Functions§
- exp_
histogram_ schema - extract_
min_ timestamp_ micros - Extract the minimum timestamp from a RecordBatch, in microseconds.
- extract_
service_ name - Extract the first service name from a RecordBatch.
- gauge_
schema - group_
batch_ by_ service - Group a RecordBatch by service_name column.
- histogram_
schema - logs_
schema - sum_
schema - to_ipc
- Serialize a RecordBatch to Arrow IPC streaming format
- to_json
- Serialize a RecordBatch to NDJSON format (newline-delimited JSON)
- traces_
schema - transform_
logs - Transform OTLP logs to Arrow RecordBatch.
- transform_
logs_ json - Transform OTLP logs to JSON values.
- transform_
logs_ partitioned - Transform OTLP logs with service-based partitioning.
- transform_
logs_ with_ observer - Transform OTLP logs while reporting phase timings to an observer.
- transform_
metrics - Transform OTLP metrics to Arrow RecordBatches.
- transform_
metrics_ json - Transform OTLP metrics to JSON values.
- transform_
metrics_ partitioned - Transform OTLP metrics with service-based partitioning.
- transform_
metrics_ with_ observer - Transform OTLP metrics while reporting phase timings to an observer.
- transform_
traces - Transform OTLP traces to Arrow RecordBatch.
- transform_
traces_ json - Transform OTLP traces to JSON values.
- transform_
traces_ partitioned - Transform OTLP traces with service-based partitioning.
- transform_
traces_ with_ observer - Transform OTLP traces while reporting phase timings to an observer.
Type Aliases§
- Result
- Result type alias using the crate’s Error type