Expand description
rypipe-core: format-agnostic ingestion framework.
The crate parses byte streams into typed Arrow record batches through a
small decoder API (Splitter + RecordParser + ColumnarSink). It has
no format-specific logic and no Python/FFI dependencies.
Re-exports§
pub use arrow_export::apply_compare_filter;pub use auto::resolve_engine;pub use auto::AutoConfig;pub use auto::EngineMode;pub use bounded::MemoryBudget;pub use bounded::MAX_SPLIT_CHUNKS;pub use budget::BudgetLedger;pub use budget::BudgetPartition;pub use budget::StreamStats;pub use consumer::BatchConsumer;pub use consumer::CollectingConsumer;pub use consumer::DiscardingConsumer;pub use decoder::find_next_record_boundary;pub use decoder::is_continued_newline;pub use decoder::split_points_to_ranges;pub use decoder::ColumnarSink;pub use decoder::RecordBoundary;pub use decoder::RecordParser;pub use decoder::Splitter;pub use engine::LocateOnly;pub use engine::TableBuilder;pub use error::Error;pub use error::Result;pub use input::InputBuffer;pub use merge::engines_to_record_batches;pub use observer::RowObserver;pub use parallel_stream::discover_schema_for_bytes;pub use parallel_stream::discovery_profile;pub use parallel_stream::reset_discovery_profile;pub use parallel_stream::ParallelStreamOpts;pub use parallel_stream::ParallelStreamingBatchIterator;pub use parallel_stream::ParallelStreamingExecutor;pub use pipeline::Pipeline;pub use plan::CompareOp;pub use plan::ExecutionPlan;pub use plan::FieldType;pub use plan::FilterPredicate;pub use plan::RegexSpec;pub use plan::TrimMode;pub use schema::DiscoveryOpts;pub use schema::FrozenSchema;pub use schema::UnknownFieldPolicy;pub use streaming::StreamingBatchIterator;pub use value::Value;
Modules§
- arrow_
export - auto
- Auto engine selection: heuristic for picking the best execution mode.
- block_
masks - BlockMasks: one AVX2/SSE2 load per 64-byte block, cached bitmasks.
- bounded
- budget
- Memory-budget partitioning, accounting, and streaming-run statistics.
- columnar
- consumer
- decoder
- dict
- engine
- error
- input
- merge
- observer
- Per-row observer hooks fired by
TableBuilderduring parsing. - parallel
- parallel_
stream - Parallel streaming executor: multi-core parsing with bounded memory.
- pipeline
- High-level, ergonomic API for running a
Splitter+RecordParserpair. - plan
- profiling
- Named stage timings for the execution engine.
- scan
- Portable scan primitives for byte searching.
- schema
- Frozen schema for parallel streaming.
- spill
- Single-pass spilling sink for bounded-memory reads.
- streaming
- value