Crate ptars 

ptars

Fast conversion from Protocol Buffers to Apache Arrow and back.

This crate is arrow-version-independent: its public API contains no arrow-rs (or prost) types. Arrow data crosses the API boundary through the Arrow C Data Interface (ffi::ArrowArray/ffi::ArrowSchema), whose ABI is frozen by the Arrow specification. You can therefore depend on ptars together with any arrow version and exchange data zero-copy.

The implementation lives in the ptars-core crate, which exposes a conventional arrow-rs-typed API if you prefer to pin the same arrow version as ptars.

Example

use ptars::{Handler, PtarsConfig};

// Descriptors are passed as serialized bytes (protoc --descriptor_set_out).
let handler = Handler::try_new(&descriptor_set_bytes, "my.Message", PtarsConfig::default())?;

// Decode serialized messages without arrow on your side at all:
let (array, schema) = handler.decode_bytes(&[Some(&message_bytes)])?;

// Import the result with *your* arrow version (any major), zero-copy.
// Both struct definitions implement the same frozen C ABI, so they can be
// transmuted into one another:
let ffi_array: arrow::ffi::FFI_ArrowArray = unsafe { std::mem::transmute(array) };
let ffi_schema: arrow::ffi::FFI_ArrowSchema = unsafe { std::mem::transmute(schema) };
let data = unsafe { arrow::ffi::from_ffi(ffi_array, &ffi_schema)? };
let batch: arrow::record_batch::RecordBatch =
    arrow::array::StructArray::from(data).into();

Modules

ffi

Hand-written Arrow C Data Interface structs.

Structs

Handler

Converts between serialized protobuf messages and Arrow record batches for a single message type, exchanging Arrow data through the C Data Interface.

PtarsConfig

Configuration for protobuf to Arrow conversions.

Enums

ConfluentWirePolicy

Policy for stripping Confluent Schema Registry wire format prefix from messages.

EnumRepr

How to represent protobuf enum fields in Arrow.

PtarsError

Error type for ptars operations.

TimeUnit

Time unit for timestamp, time of day and duration values.