Lightstream

High-throughput, composable table transport for Arrow-compatible data. Move tables between processes, services and storage without adopting gRPC or writing transport-specific framing.

[dependencies]
lightstream = { version = "0.5.0", features = ["tcp", "mmap", "zstd"] }

Quick start

Receiver

use futures_util::StreamExt;
use lightstream::models::readers::tcp::TcpTableReader;

let mut reader = TcpTableReader::connect("127.0.0.1:9000").await?;

while let Some(result) = reader.next().await {
    let table = result?;
    process(table);
}

Sender

use lightstream::models::writers::tcp::TcpTableWriter;

let mut writer = TcpTableWriter::connect("127.0.0.1:9000", schema, None).await?;
writer.write_table(batch_1).await?;
writer.finish().await?;

Other transports (HTTP/2, QUIC, WebSocket, UDS, files, memory maps) expose the same table-oriented interface.

Why

• One model, many transports. Identical table APIs across TCP, HTTP/2, QUIC, WebSocket, Unix sockets, files, memory maps and chunked datasets.
• Fast paths preserved. Minarrow's 64-byte-aligned buffers flow through supported transports zero-copy.
• Composable layers. Encoding, framing, buffering and transport remain independent — use complete readers/writers or assemble lower-level codecs directly.
• Practical optionality. TLS, compression (zstd/snappy), decode limits for untrusted input, Linux io_uring, minimal dependencies.

Performance

Lightstream typically outperforms Arrow-rs file readers by ~30% and Arrow Flight by ~60% on cross-host benchmarks. See benches/README.md for detailed methodology.

Capabilities

Transports: TCP, Unix domain sockets, HTTP/2, QUIC, WebSocket, WebTransport, standard I/O, io_uring.

Formats: Arrow IPC (file and stream), CSV, JSON, NDJSON, Parquet, TLV framing, memory maps, chunked datasets.

Protocol: Lightstream message multiplexing for Arrow tables, Protobuf and MessagePack over one connection.

Advanced: Parallel streams across TCP/QUIC/HTTP/2 with optional global ordering, compression, TLS, configurable decode safety.

Common patterns

Compression

Compress tables before sending or writing to disk.

use lightstream::compression::Compression;

let writer = TcpTableWriter::connect(
    address,
    schema,
    Some(Compression::Zstd),
).await?;

Memory-mapped Arrow IPC

The mmap reader is very fast. Warm mmap rivals standalone RAM speed, as it gets out the way. Try the benchmark.

use lightstream::models::readers::ipc::mmap_table::MmapTableReader;

let reader = MmapTableReader::open("data.arrow")?;
for i in 0..reader.num_batches() {
    let table = reader.read_batch(i)?;
    process(table);
}

Lightstream protocol (multiplex protobuf messages and arrow tables)

use lightstream::models::protocol::connection::TcpLightstreamConnection;
use lightstream::models::protocol::LightstreamMessage;

let mut conn = TcpLightstreamConnection::from_tcp(stream);
conn.register_message("event");
conn.register_table("metrics", schema);

conn.send("event", b"user-login").await?;
conn.send_table("metrics", &table).await?;

while let Some(msg) = conn.recv().await {
    match msg? {
        // Protobuf message
        LightstreamMessage::Message { tag, payload } => { /* … */ }
        // Arrow table
        LightstreamMessage::Table { table, .. } => { /* … */ }
    }
}

Parallel streams

TCP, QUIC and HTTP/2 readers and writers distribute tables across multiple connections, aggregating throughput beyond single-flow limits. Readers can restore global send order or return items in arrival order.

The Lightstream protocol automatically provides total ordering across fanned out parallel streams. See the benchmarks for an example.

TLS

Enable the tls feature for TCP, WebSocket and HTTP/2. QUIC and WebTransport include TLS at the protocol level.

let writer = TcpTableWriter::connect_tls(
    "api.example.com:9443",
    server_name,
    config,
    schema,
    None,
).await?;

Examples

cargo run --example tcp_arrow --features tcp
cargo run --example websocket_arrow --features websocket
cargo run --example http_arrow --features http
cargo run --example quic_arrow --features quic
cargo run --example uds_arrow --features uds

Feature flags

Buffering and safety

Streaming decoders use a per-stream arena reserving 2 GiB of virtual address space (configurable via LIGHTSTREAM_ARENA_CAPACITY). Physical memory is committed only as data arrives. Decode paths validate all input and enforce configurable limits on frame size, schema dimensions, strings, allocations and decompressed data.

Architecture

Applications can use complete readers and writers or assemble lower-level components directly.

Licence

Mozilla Public License 2.0. © 2025–2026 Peter Garfield Bower.

Maintained by SpaceCell.