rypipe-core
Format-agnostic columnar ingestion engine. Parses row-oriented byte streams into Apache Arrow record batches with parallel scheduling, memory-bounded execution, and query pushdown.
This crate is the pure Rust engine: no Python, no FFI, and no format-specific logic. It ships no parsers. Formats such as XML, JSON, CSV, and HTML live in separate adapter crates.
Part of the rypipe project. The
rypipe Python package wraps this engine through PyO3.
What it does
The engine separates format-specific parsing (splitting, row extraction) from format-agnostic execution (typed column builders, projection, filtering, dictionary encoding, parallel scheduling, memory-bounded execution, and Arrow export).
Adding a format means implementing two traits, Splitter and RecordParser.
An adapter answers two questions, where are the rows and what are the fields,
and the engine handles the rest.
Usage
[]
= "0.5"
use ;
use ; // separate adapter crate
let batch = new
.with_plan
.read_path?;
Features
- Zero-copy friendly: decoders emit borrowed strings; the engine copies only when necessary.
- Parallel by default: chunked parsing with
rayonscales to many cores. - Memory bounded: stream files larger than RAM with a configurable budget.
- Typed columns: cast strings to
int64,float64, orboolduring parse. - Pushdown: rename, drop, type, and filter rows while parsing.
- Dictionary encoding: explicit or automatic low-cardinality encoding.
- Arrow native: produces
RecordBatchand exports via the C Data Interface.
Cargo features
| Feature | Effect |
|---|---|
mmap |
Memory-mapped file input via memmap2 |
Public API
Pipeline, ExecutionPlan, TableBuilder, Value, InputBuffer,
MemoryBudget, FieldType, CompareOp, FilterPredicate,
engines_to_record_batches, and apply_compare_filter. The decoder API is
Splitter, RecordParser, and ColumnarSink.
Arrow and PyO3 version pins
arrow is pinned exactly (=59.3.0). The Arrow C Data Interface is not stable
across minor versions, and this crate exports through it. Bump only after
testing both directions.
rypipe-python pins pyo3 to the version required by arrow-pyarrow
(currently 0.29). The two must move together; bumping one without the other
produces a native python link conflict in Cargo.
Documentation
Full guides live at rypipe.emiliano-go.com, including architecture, the Rust API, and writing a format adapter.
License
MIT