Skip to main content

Crate oxidelake_storage

Crate oxidelake_storage 

Source
Expand description

Storage layer (docs/SPEC.md §2.4): Parquet writer and pruning configuration, Arrow IPC spill/cache files, and the object-store IO layer.

There is no private file format (ADR-0014). Lake data is Parquet read through DataFusion, whose statistics, page-index and Bloom-filter pruning this crate configures and proves (see scan_pruning_metrics); spill and cache files are Arrow IPC with 64-byte-aligned buffers; all IO goes through object_store::ObjectStore — LocalFileSystem by default, the io_uring implementation behind the io-uring feature on Linux.

Dependency direction: depends on oxidelake-core and oxidelake-memory.

Re-exports§

pub use datagen::CHUNK_ROWS;
pub use datagen::DemoOptions;
pub use datagen::DemoTable;
pub use datagen::SPECIAL_FLOAT_EVERY;
pub use datagen::SPECIAL_FLOATS;
pub use datagen::demo_batches;
pub use datagen::demo_batches_with;
pub use datagen::demo_chunks;
pub use datagen::demo_chunks_with;
pub use datagen::demo_schema;
pub use datagen::demo_write_options;
pub use datagen::write_demo_table;
pub use datagen::write_demo_table_with;
pub use errors::classify_error;
pub use ipc::read_ipc_file;
pub use ipc::write_ipc_file;
pub use parquet_io::ParquetFileWriter;
pub use parquet_io::write_parquet;
pub use parquet_opts::Compression;
pub use parquet_opts::ParquetWriteOptions;
pub use scan_guard::ParquetScanGuard;
pub use session::GPU_BATCH_SIZE;
pub use session::PruningSummary;
pub use session::pruning_session;
pub use session::register_parquet_table;
pub use session::scan_pruning_metrics;
pub use session::with_gpu_batch_size;
pub use session::with_pruning;
pub use session::with_pruning_disabled;
pub use store::BoundedReadStore;
pub use store::default_object_store;
pub use store::register_local_store;
pub use uring::UringLocalFileSystem;

Modules§

datagen
Deterministic demo-dataset generator behind oxide gen-data.
errors
Maps DataFusion/Parquet/object-store failures onto EngineError at the storage boundary: corrupt or truncated files are Format, IO failures are Io, our own errors come back unchanged.
ipc
Arrow IPC files for spill and hot caches: raw Arrow buffers, 64-byte aligned, no compression, so batches come back without decode.
parquet_io
Writing Parquet files with OxideLake’s writer settings.
parquet_opts
Parquet writer configuration tuned for OxideLake datasets.
scan_guard
Checks a Parquet file’s metadata when a scan opens it, not only when its table was registered (#109).
session
Session configuration for Parquet pruning, table registration, and the proof that pruning actually happened (read back from the scan’s metrics).
store
Object-store construction and registration.
uring
io_uring-backed local object store (feature io-uring, Linux only).