Skip to main content

Crate rypipe_core

Crate rypipe_core 

Source
Expand description

rypipe-core: format-agnostic ingestion framework.

The crate parses byte streams into typed Arrow record batches through a small decoder API (Splitter + RecordParser + ColumnarSink). It has no format-specific logic and no Python/FFI dependencies.

Re-exports§

pub use arrow_export::apply_compare_filter;
pub use auto::resolve_engine;
pub use auto::AutoConfig;
pub use auto::EngineMode;
pub use bounded::MemoryBudget;
pub use bounded::MAX_SPLIT_CHUNKS;
pub use budget::BudgetLedger;
pub use budget::BudgetPartition;
pub use budget::StreamStats;
pub use consumer::BatchConsumer;
pub use consumer::CollectingConsumer;
pub use consumer::DiscardingConsumer;
pub use decoder::find_next_record_boundary;
pub use decoder::is_continued_newline;
pub use decoder::split_points_to_ranges;
pub use decoder::ColumnarSink;
pub use decoder::RecordBoundary;
pub use decoder::RecordParser;
pub use decoder::Splitter;
pub use engine::LocateOnly;
pub use engine::TableBuilder;
pub use error::Error;
pub use error::Result;
pub use input::InputBuffer;
pub use merge::engines_to_record_batches;
pub use observer::RowObserver;
pub use parallel_stream::discover_schema_for_bytes;
pub use parallel_stream::discovery_profile;
pub use parallel_stream::reset_discovery_profile;
pub use parallel_stream::ParallelStreamOpts;
pub use parallel_stream::ParallelStreamingBatchIterator;
pub use parallel_stream::ParallelStreamingExecutor;
pub use pipeline::Pipeline;
pub use plan::CompareOp;
pub use plan::ExecutionPlan;
pub use plan::FieldType;
pub use plan::FilterPredicate;
pub use plan::RegexSpec;
pub use plan::TrimMode;
pub use schema::DiscoveryOpts;
pub use schema::FrozenSchema;
pub use schema::UnknownFieldPolicy;
pub use streaming::StreamingBatchIterator;
pub use value::Value;

Modules§

arrow_export
auto
Auto engine selection: heuristic for picking the best execution mode.
block_masks
BlockMasks: one AVX2/SSE2 load per 64-byte block, cached bitmasks.
bounded
budget
Memory-budget partitioning, accounting, and streaming-run statistics.
columnar
consumer
decoder
dict
engine
error
input
merge
observer
Per-row observer hooks fired by TableBuilder during parsing.
parallel
parallel_stream
Parallel streaming executor: multi-core parsing with bounded memory.
pipeline
High-level, ergonomic API for running a Splitter + RecordParser pair.
plan
profiling
Named stage timings for the execution engine.
scan
Portable scan primitives for byte searching.
schema
Frozen schema for parallel streaming.
spill
Single-pass spilling sink for bounded-memory reads.
streaming
value