Skip to main content

Crate rypipe_core

Crate rypipe_core 

Source
Expand description

rypipe-core: format-agnostic ingestion framework.

The crate parses byte streams into typed Arrow record batches through a small decoder API (Splitter + RecordParser + ColumnarSink). It has no format-specific logic and no Python/FFI dependencies.

Re-exports§

pub use arrow_export::apply_compare_filter;
pub use bounded::MemoryBudget;
pub use consumer::BatchConsumer;
pub use consumer::CollectingConsumer;
pub use consumer::DiscardingConsumer;
pub use decoder::split_points_to_ranges;
pub use decoder::ColumnarSink;
pub use decoder::RecordParser;
pub use decoder::Splitter;
pub use engine::LocateOnly;
pub use engine::TableBuilder;
pub use error::Error;
pub use error::Result;
pub use input::InputBuffer;
pub use merge::engines_to_record_batches;
pub use parallel_stream::discover_schema_for_bytes;
pub use parallel_stream::discovery_profile;
pub use parallel_stream::reset_discovery_profile;
pub use parallel_stream::ParallelStreamOpts;
pub use parallel_stream::ParallelStreamingBatchIterator;
pub use parallel_stream::ParallelStreamingExecutor;
pub use pipeline::Pipeline;
pub use plan::CompareOp;
pub use plan::ExecutionPlan;
pub use plan::FieldType;
pub use plan::FilterPredicate;
pub use schema::DiscoveryOpts;
pub use schema::FrozenSchema;
pub use schema::UnknownFieldPolicy;
pub use streaming::StreamingBatchIterator;
pub use value::Value;

Modules§

arrow_export
block_masks
BlockMasks: one AVX2/SSE2 load per 64-byte block, cached bitmasks.
bounded
columnar
consumer
decoder
dict
engine
error
input
merge
parallel
parallel_stream
Parallel streaming executor: multi-core parsing with bounded memory.
pipeline
High-level, ergonomic API for running a Splitter + RecordParser pair.
plan
scan
Portable scan primitives for byte searching.
schema
Frozen schema for parallel streaming.
streaming
value