otlp2records

Transform OTLP and native OTAP telemetry (logs, traces, metrics) into Arrow RecordBatches.

A high-performance, WASM-compatible library for converting OpenTelemetry Protocol (OTLP) data to Apache Arrow format for efficient storage and querying.

Currently consumed by duckdb-otlp, otlp2parquet and otlp2pipeline.

Design Principles

• No I/O: Core never touches network or filesystem
• No async: Pure synchronous transforms
• WASM-first: All dependencies compile to wasm32
• Arrow-native: RecordBatch is the canonical output format
• Shared telemetry normalization: OTLP and native OTAP logs, traces, and metrics converge through semantic view traits compatible with OpenTelemetry Arrow pdata-views

Features

• Transform OTLP logs, traces, and metrics to Arrow RecordBatches
• Decode native OTAP logs, traces, and univariate metrics to the same normalized Arrow outputs
• Support for both Protobuf and JSON input formats
• Output to NDJSON, Arrow IPC, or Parquet
• Direct OTLP-to-Arrow hot path for high-throughput ingestion
• JSON/JSONL support through OTLP request normalization into the same Arrow builders

Installation

Add to your Cargo.toml:

[dependencies]
otlp2records = "0.8"

# Optional: Enable Parquet output
otlp2records = { version = "0.8", features = ["parquet"] }

# Optional: Enable WASM bindings
otlp2records = { version = "0.8", features = ["wasm"] }

Usage

Rust API

High-level API (Recommended)

use otlp2records::{transform_logs, transform_traces, transform_metrics, InputFormat};

// Transform OTLP logs
let bytes: &[u8] = /* OTLP log data */;
let batch = transform_logs(bytes, InputFormat::Protobuf)?;
println!("Transformed {} log records", batch.num_rows());

// Transform OTLP traces
let batch = transform_traces(bytes, InputFormat::Json)?;
println!("Transformed {} spans", batch.num_rows());

// Transform OTLP metrics (returns separate batches by type)
let batches = transform_metrics(bytes, InputFormat::Protobuf)?;
if let Some(gauge) = batches.gauge {
    println!("Transformed {} gauge metrics", gauge.num_rows());
}
if let Some(sum) = batches.sum {
    println!("Transformed {} sum metrics", sum.num_rows());
}

Output Formats

use otlp2records::{transform_logs, to_json, to_ipc, InputFormat};

let batch = transform_logs(bytes, InputFormat::Protobuf)?;

// Output as NDJSON
let ndjson: Vec<u8> = to_json(&batch)?;

// Output as Arrow IPC (streaming format)
let ipc: Vec<u8> = to_ipc(&batch)?;

// Output as Parquet (requires "parquet" feature)
#[cfg(feature = "parquet")]
let parquet: Vec<u8> = otlp2records::to_parquet(&batch)?;

WASM Usage

Build with the wasm feature for browser/Node.js environments:

cargo build --target wasm32-unknown-unknown --features wasm

Crates.io consumers should select the getrandom 0.3 browser backend in the top-level workspace (dependency-local Cargo config is not inherited):

# .cargo/config.toml
[target.wasm32-unknown-unknown]
rustflags = ['--cfg', 'getrandom_backend="wasm_js"']

import init, { transform_logs_wasm } from './otlp2records.js';

await init();

// Transform OTLP logs (Uint8Array) to Arrow IPC
const otlpBytes = new Uint8Array(/* ... */);
const arrowIpc = transform_logs_wasm(otlpBytes, "protobuf");

API Overview

Input Formats

High-level Functions

Breaking Changes In 0.8.0

The 0.7 to 0.8 release intentionally changes the default normalized schema. The existing transform_logs, transform_traces, and transform_metrics APIs still return flattened batches by default, but downstream code that selects columns by name or expects specific Arrow physical types must be updated.

Key normalized-schema changes:

• OTLP/OTAP field names replace older ClickStack-style names: for example, timestamp becomes time_unix_nano for logs and metrics, trace timestamp becomes start_time_unix_nano, span_name becomes name, span_kind becomes kind, and metric metric_name/metric_description/ metric_unit become name/description/unit.
• Timestamps now use Arrow Timestamp(Nanosecond) instead of microsecond or millisecond-scaled integer columns. Span duration is duration_time_unix_nano with Arrow Duration(Nanosecond).
• Trace and span identifiers are Arrow FixedSizeBinary(16) and FixedSizeBinary(8) instead of hex strings.
• Metric number values are split into nullable int_value and double_value columns instead of a single Float64 value column.
• Histogram bucket columns now use typed Arrow list columns instead of JSON strings, and dropped counts/flags/count fields use unsigned Arrow integer types where OTAP does.

The flattened JSON convenience columns remain for now: resource_attributes, scope_attributes, signal attribute JSON columns, events_json, links_json, and exemplars_json.

Transform Observation

Production callers can opt into phase timings and counters without changing output semantics:

Implement TransformObserver to receive TransformPhaseTiming and TransformCounterValue events. Counters include duplicate resource/scope context hits and misses plus repeated resource/scope attribute row-copy counts and bytes.

Output Functions

These serializers operate on one RecordBatch at a time.

Schemas

Architecture

                              +-------------------+
                              |   OTLP Input      |
                              | (Protobuf / JSON) |
                              +---------+---------+
                                        |
                                        v
                              +---------+---------+
                              |   Format Dispatch |
                              | (protobuf/jsonl)  |
                              +---------+---------+
                                        |
                                        v
                              +---------+---------+
                              | OTLP Request      |
                              | (prost structs)   |
                              +---------+---------+
                                        |
                                        v
                              +---------+---------+
                              | Arrow Builders    |
                              | (direct columns)  |
                              +---------+---------+
                                        |
                                        v
                              +---------+---------+
                              |   RecordBatch     |
                              +---------+---------+
                                        |
                  +---------------------+---------------------+
                  |                     |                     |
                  v                     v                     v
          +-------+-------+     +-------+-------+     +-------+-------+
          |    NDJSON     |     |   Arrow IPC   |     |    Parquet    |
          +---------------+     +---------------+     +---------------+

Public Surface

• transform functions: Convert OTLP logs, traces, and metrics to Arrow batches
• schema functions: Return the Arrow schemas used by the transform functions
• partition helpers: Group transformed batches by service
• output helpers: Serialize RecordBatches to NDJSON, Arrow IPC, or Parquet
• wasm: WASM bindings (optional)

Output Schemas

The transform APIs emit a flattened, normalized schema. In 0.8.0 it uses OTAP-compatible field names and high-value Arrow physical types while keeping the flattened resource/scope/attribute convenience columns.

Logs Schema

Traces Schema

Gauge Metrics Schema

Sum Metrics Schema

Includes all gauge fields plus:

Histogram Metrics Schema

Histogram metrics use the common metric context fields above, plus count (UInt64), sum, min, max, typed bucket_counts (List<UInt64>), typed explicit_bounds (List<Float64>), flags, exemplars_json, and aggregation_temporality.

Exponential Histogram Metrics Schema

Exponential histograms use the common metric context fields above, plus count (UInt64), sum, min, max, scale, zero_count (UInt64), zero_threshold, typed positive/negative bucket-count lists, flags, exemplars_json, and aggregation_temporality.

Cargo Features

Native OTAP Input

Canonical BatchArrowRecords logs, traces, and univariate metrics decode through a stateful OtapDecoder into the existing normalized schemas. Uncompressed and LZ4 Arrow IPC decode on every target, including WASM. The upstream Producer defaults to Zstandard; enable otap-zstd to accept it on native targets. That feature uses Arrow's bundled C backend, so it is native-only — combining it with a wasm32 target is unsupported and will not build. See docs/otap-input.md for the API, protocol coverage, provenance, and remaining protocol boundaries.

Performance

• Transforms are plain Rust functions with no interpreter or runtime overhead
• Arc-shared resource/scope values reduce memory allocations
• Arrow columnar format enables efficient compression
• Release builds use LTO and size optimization

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contributing

Contributions welcome! Please ensure:

1. All tests pass: cargo test
2. Code is formatted: cargo fmt
3. No clippy warnings: cargo clippy -- -D warnings