Struct LakeScanner 

pub struct LakeScanner {
    pub paths: Vec<String>,
}

Multi-format lake scanner.

Fields

paths: Vec<String>

Implementations

impl LakeScanner

pub fn new(paths: Vec<String>) -> Self

pub fn from_request(req: &ScanRequest) -> Self

pub fn list_files(&self, req: &ScanRequest) -> Result<(PathBuf, Vec<PathBuf>)>

Resolve scan root and list bronze files after partition/glob filters.

pub async fn scan(&self, req: &ScanRequest) -> Result<Vec<RecordBatch>>

Scan using a full ScanRequest (format-aware). Loads all batches into memory.

pub fn scan_spill_to_parquet( &self, req: &ScanRequest, dest: &Path, opts: &MaterializeWriteOptions, ) -> Result<StreamWriteStats>

Stream bronze files into a single Parquet file (atomic publish).

Peak memory ≈ one source file’s batches + Parquet encoder — not the full hive tree. Used for Arrow IPC multi-file bronze when scan.spill_arrow_ipc is true.

pub async fn scan_path( &self, path: impl AsRef<Path>, schema: SchemaRef, ) -> Result<Vec<RecordBatch>>

Legacy helper: recurse path and read by extension with an explicit schema (used by unit tests and jshift-projected JSONL).