Crate otlp2records 

otlp2records - Transform OTLP telemetry to Arrow RecordBatches

This crate provides synchronous, WASM-compatible transformation of OpenTelemetry Protocol (OTLP) data (logs, traces, metrics) into Arrow RecordBatches.

Design Principles

• No I/O: Core never touches network or filesystem
• No async: Pure synchronous transforms
• WASM-first: All dependencies compile to wasm32
• Arrow-native: RecordBatch is the canonical output format

High-level API

The simplest way to use this crate is with the high-level transform functions:

use otlp2records::{transform_logs, transform_traces, transform_metrics, InputFormat};

// Transform OTLP logs to Arrow RecordBatch
let batch = transform_logs(bytes, InputFormat::Protobuf)?;

// Transform OTLP traces to Arrow RecordBatch
let batch = transform_traces(bytes, InputFormat::Json)?;

// Transform OTLP metrics to Arrow RecordBatches (separate gauge and sum)
let batches = transform_metrics(bytes, InputFormat::Protobuf)?;
if let Some(gauge_batch) = batches.gauge {
    // Process gauge metrics
}
if let Some(sum_batch) = batches.sum {
    // Process sum metrics
}

Lower-level API

For more control over individual transformation steps:

use otlp2records::{decode_logs, apply_log_transform, values_to_arrow, logs_schema, InputFormat};

// Step 1: Decode OTLP bytes to VRL Values
let values = decode_logs(bytes, InputFormat::Protobuf)?;

// Step 2: Apply VRL transformation
let transformed = apply_log_transform(values)?;

// Step 3: Convert to Arrow RecordBatch
let batch = values_to_arrow(&transformed, &logs_schema())?;

Re-exports

pub use arrow::exp_histogram_schema;

pub use arrow::extract_min_timestamp_micros;

pub use arrow::extract_service_name;

pub use arrow::gauge_schema;

pub use arrow::group_batch_by_service;

pub use arrow::histogram_schema;

pub use arrow::logs_schema;

pub use arrow::sum_schema;

pub use arrow::traces_schema;

pub use arrow::values_to_arrow;

pub use arrow::PartitionedBatch;

pub use arrow::PartitionedMetrics;

pub use arrow::ServiceGroupedBatches;

pub use decode::count_skipped_metric_data_points;

pub use decode::decode_logs;

pub use decode::decode_metrics;

pub use decode::decode_traces;

pub use decode::normalise_json_value;

pub use decode::normalize_json_bytes;

pub use decode::DecodeMetricsResult;

pub use decode::InputFormat;

pub use decode::MetricSkipCounts;

pub use decode::SkippedMetrics;

pub use error::Error;

pub use error::Result;

pub use output::to_ipc;

pub use output::to_json;

pub use schemas::schema_def;

pub use schemas::schema_defs;

pub use schemas::SchemaDef;

pub use schemas::SchemaField;

pub use transform::VrlError;

pub use transform::VrlTransformer;

pub use transform::OTLP_EXP_HISTOGRAM_PROGRAM;

pub use transform::OTLP_GAUGE_PROGRAM;

pub use transform::OTLP_HISTOGRAM_PROGRAM;

pub use transform::OTLP_LOGS_PROGRAM;

pub use transform::OTLP_SUM_PROGRAM;

pub use transform::OTLP_TRACES_PROGRAM;

Modules

arrow

Arrow layer for otlp2records

convert

Shared conversion utilities for VRL Value to JSON

decode

OTLP decode layer - transforms raw bytes into VRL Values

error

Error types for otlp2records

output

Output serialization for Arrow RecordBatches

schemas

Schema definitions parsed from VRL @schema annotations.

transform

VRL transformation module for OTLP data

Structs

JsonMetricBatches

Result of transforming OTLP metrics to JSON values.

MetricBatches

Result of transforming OTLP metrics to Arrow RecordBatches.

MetricValues

Result of applying VRL transformation to metrics.

Functions

apply_log_transform

Apply VRL transformation to decoded log values.

apply_metric_transform

Apply VRL transformation to decoded metric values.

apply_trace_transform

Apply VRL transformation to decoded trace values.

transform_logs

Transform OTLP logs to Arrow RecordBatch.

transform_logs_json

Transform OTLP logs to JSON values.

transform_logs_partitioned

Transform OTLP logs with service-based partitioning.

transform_metrics

Transform OTLP metrics to Arrow RecordBatches.

transform_metrics_json

Transform OTLP metrics to JSON values.

transform_metrics_partitioned

Transform OTLP metrics with service-based partitioning.

transform_traces

Transform OTLP traces to Arrow RecordBatch.

transform_traces_json

Transform OTLP traces to JSON values.

transform_traces_partitioned

Transform OTLP traces with service-based partitioning.