Expand description
Storage layer (docs/SPEC.md §2.4): Parquet writer and pruning configuration, Arrow IPC spill/cache files, and the object-store IO layer.
There is no private file format (ADR-0014). Lake data is Parquet read
through DataFusion, whose statistics, page-index and Bloom-filter pruning
this crate configures and proves (see scan_pruning_metrics); spill and
cache files are Arrow IPC with 64-byte-aligned buffers; all IO goes through
object_store::ObjectStore — LocalFileSystem by default, the io_uring
implementation behind the io-uring feature on Linux.
Dependency direction: depends on oxidelake-core and oxidelake-memory.
Re-exports§
pub use datagen::CHUNK_ROWS;pub use datagen::DemoOptions;pub use datagen::DemoTable;pub use datagen::SPECIAL_FLOAT_EVERY;pub use datagen::SPECIAL_FLOATS;pub use datagen::demo_batches;pub use datagen::demo_batches_with;pub use datagen::demo_chunks;pub use datagen::demo_chunks_with;pub use datagen::demo_schema;pub use datagen::demo_write_options;pub use datagen::write_demo_table;pub use datagen::write_demo_table_with;pub use errors::classify_error;pub use ipc::read_ipc_file;pub use ipc::write_ipc_file;pub use parquet_io::ParquetFileWriter;pub use parquet_io::write_parquet;pub use parquet_opts::Compression;pub use parquet_opts::ParquetWriteOptions;pub use scan_guard::ParquetScanGuard;pub use session::GPU_BATCH_SIZE;pub use session::PruningSummary;pub use session::pruning_session;pub use session::register_parquet_table;pub use session::scan_pruning_metrics;pub use session::with_gpu_batch_size;pub use session::with_pruning;pub use session::with_pruning_disabled;pub use store::BoundedReadStore;pub use store::default_object_store;pub use store::register_local_store;pub use uring::UringLocalFileSystem;
Modules§
- datagen
- Deterministic demo-dataset generator behind
oxide gen-data. - errors
- Maps DataFusion/Parquet/object-store failures onto
EngineErrorat the storage boundary: corrupt or truncated files areFormat, IO failures areIo, our own errors come back unchanged. - ipc
- Arrow IPC files for spill and hot caches: raw Arrow buffers, 64-byte aligned, no compression, so batches come back without decode.
- parquet_
io - Writing Parquet files with OxideLake’s writer settings.
- parquet_
opts - Parquet writer configuration tuned for OxideLake datasets.
- scan_
guard - Checks a Parquet file’s metadata when a scan opens it, not only when its table was registered (#109).
- session
- Session configuration for Parquet pruning, table registration, and the proof that pruning actually happened (read back from the scan’s metrics).
- store
- Object-store construction and registration.
- uring
- io_uring-backed local object store (feature
io-uring, Linux only).