Expand description
rypipe-core: format-agnostic ingestion framework.
The crate parses byte streams into typed Arrow record batches through a
small decoder API (Splitter + RecordParser + ColumnarSink). It has
no format-specific logic and no Python/FFI dependencies.
Re-exports§
pub use arrow_export::apply_compare_filter;pub use bounded::MemoryBudget;pub use consumer::BatchConsumer;pub use consumer::CollectingConsumer;pub use consumer::DiscardingConsumer;pub use decoder::split_points_to_ranges;pub use decoder::ColumnarSink;pub use decoder::RecordParser;pub use decoder::Splitter;pub use engine::LocateOnly;pub use engine::TableBuilder;pub use error::Error;pub use error::Result;pub use input::InputBuffer;pub use merge::engines_to_record_batches;pub use parallel_stream::discover_schema_for_bytes;pub use parallel_stream::discovery_profile;pub use parallel_stream::reset_discovery_profile;pub use parallel_stream::ParallelStreamOpts;pub use parallel_stream::ParallelStreamingBatchIterator;pub use parallel_stream::ParallelStreamingExecutor;pub use pipeline::Pipeline;pub use plan::CompareOp;pub use plan::ExecutionPlan;pub use plan::FieldType;pub use plan::FilterPredicate;pub use schema::DiscoveryOpts;pub use schema::FrozenSchema;pub use schema::UnknownFieldPolicy;pub use streaming::StreamingBatchIterator;pub use value::Value;
Modules§
- arrow_
export - block_
masks - BlockMasks: one AVX2/SSE2 load per 64-byte block, cached bitmasks.
- bounded
- columnar
- consumer
- decoder
- dict
- engine
- error
- input
- merge
- parallel
- parallel_
stream - Parallel streaming executor: multi-core parsing with bounded memory.
- pipeline
- High-level, ergonomic API for running a
Splitter+RecordParserpair. - plan
- scan
- Portable scan primitives for byte searching.
- schema
- Frozen schema for parallel streaming.
- streaming
- value