Crate lightstream

Crate lightstream 

Source
Expand description

§Lightstream — Streaming Arrow IPC, TLV, and Parquet I/O for Minarrow

Lightstream provides composable building blocks for high-performance data I/O in Rust.

It extends Minarrow with a set of modular, format-aware components for:

  • High-performance asynchronous Arrow IPC streaming and file writing
  • Framed decoders and sinks for IPC, TLV, CSV, and opt-in Parquet
  • Zero-Copy memory-mapped Arrow file reads
  • Direct Tokio integration with zero-copy buffers
  • 64-byte SIMD aligned readers and writers (the only Arrow crate that provides this in 2025)

§Design Principles

  • Customisable - You own the buffer – No hidden buffering or lifecycle surprises. All streaming is pull-based or sink-driven.
  • Composable - Layerable codecs – Each encoder, decoder, sink, and stream adapter is layerable, and your bytestream propagates up.
  • Control - Wire-level framing – Arrow IPC, TLV, CSV, and Parquet handled at the transport boundary, not fused into business logic.
  • Power - 64-byte aligned by default – All buffers use 64-byte aligned memory via [Vec64] for deterministic SIMD - not re-allocating during hotloop calculations where you need it fast.
  • Extensible - all primitives are provided to create your own data wire formats, and customise it to your stack. We also welcome contributions.

§Highlights

  • ✅ Fully async-compatible with tokio::io::AsyncWrite
  • ✅ Pluggable encoders and frame formats
  • ✅ Arrow IPC framing with dictionary + schema support
  • ✅ Categorical dictionary support
  • ✅ Compatible with minarrow::Table and minarrow::SuperTable
  • ✅ Optional support for parquet, zstd, snappy, and mmap

§Example — Arrow Table Writer

use minarrow::{arr_i32, arr_str32, vec64, FieldArray, Table};
use lightstream::models::writers::ipc::table_writer::TableWriter;
use lightstream::enums::IPCMessageProtocol;
use tokio::fs::File;

let col1 = FieldArray::from_inner("ids", arr_i32![1, 2, 3]);
let col2 = FieldArray::from_inner("names", arr_str32!["a", "b", "c"]);
let table = Table::new("example".to_string(), vec![col1, col2].into());

let schema: Vec<_> = table.schema().iter().map(|f| (**f).clone()).collect();
let file = File::create("out.arrow").await?;

let mut writer = TableWriter::new(file, schema, IPCMessageProtocol::File)?;
writer.write_table(table).await?;
writer.finish().await?;

See the README for more examples.

Re-exports§

pub use crate::arrow::message::org::apache::arrow::flatbuf::Message as AFMessage;
pub use crate::arrow::message::org::apache::arrow::flatbuf::MessageHeader as AFMessageHeader;

Modules§

arrow
FlatBuffers-compiled Arrow IPC metadata support.
compression
Compression options and helpers. Compression utilities for parquet_writer.
constants
Shared protocol constants.
enums
Internal enums for decode results, protocol kinds, etc.
error
Crate-wide error type.
models
Codec implementations, readers, writers, and I/O models
traits
Composable traits for streaming bytes and frames.
utils
Utility helpers Contains helper utilities reused across Lightstream

Macros§

debug_println