Expand description
§Lightstream
High-throughput, composable table transport for Arrow-compatible data. Move
minarrow tables between processes,
services and storage without adopting gRPC or writing transport-specific
framing.
One table-oriented reader and writer interface spans every transport and format. Encoding, framing, buffering and transport stay independent, so an application uses the complete readers and writers or composes the lower-level codecs itself.
§Capabilities
- Transports: TCP, Unix domain sockets, HTTP/2, QUIC, WebSocket,
WebTransport, standard I/O and Linux
io_uring. - Formats: Arrow IPC file and stream, CSV, JSON and NDJSON, Parquet, TLV framing, memory-mapped reads and chunked datasets.
- Protocol: the Lightstream protocol multiplexes Arrow tables, Protobuf and MessagePack over one connection, with optional global ordering across parallel streams.
- Optionality: TLS, zstd and Snappy compression, and configurable decode limits for untrusted input.
Minarrow’s 64-byte-aligned buffers flow through the supported transports zero-copy, so the SIMD-friendly layout survives the trip to the wire or disk. Warm memory-mapped reads approach RAM speed.
§Feature flags
Cargo features gate the optional formats and transports, including tcp,
uds, http, quic, websocket, stdio, mmap, csv, json,
parquet, zstd, snappy, protocol and io_uring. See the project
README for the full matrix and
build configuration.
§Example: writing an Arrow IPC file
use lightstream::enums::IPCMessageProtocol;
use lightstream::models::writers::ipc::table::TableWriter;
use minarrow::{FieldArray, Table, arr_i32, arr_str32};
use tokio::fs::File;
let ids = FieldArray::from_arr("ids", arr_i32![1, 2, 3]);
let names = FieldArray::from_arr("names", arr_str32!["a", "b", "c"]);
let table = Table::new("example".to_string(), vec![ids, names].into());
let file = File::create("out.arrow").await?;
let mut writer =
TableWriter::from_schema(file, table.schema(), IPCMessageProtocol::File)?;
writer.write_table(table).await?;
writer.finish().await?;See the project README for build configuration and further examples.
Re-exports§
pub use crate::arrow::message::org::apache::arrow::flatbuf::Message as AFMessage;pub use crate::arrow::message::org::apache::arrow::flatbuf::MessageHeader as AFMessageHeader;
Modules§
- arrow
- FlatBuffers-compiled Arrow IPC metadata support.
- compression
- Compression options and helpers. Compression codecs for Arrow IPC and Parquet.
- constants
- Shared protocol constants.
- enums
- Internal enums for decode results, protocol kinds, etc.
- error
- Crate-wide error type.
- models
- Codec implementations, readers, writers, and I/O models
- traits
- Composable traits for streaming bytes and frames.
- utils
- Utility helpers Contains helper utilities reused across Lightstream