Struct ScanMetadata 

pub struct ScanMetadata {
    pub scan_files: FilteredEngineData,
    pub scan_file_transforms: Vec<Option<ExpressionRef>>,
}

ScanMetadata contains (1) a batch of FilteredEngineData specifying data files to be scanned and (2) a vector of transforms (one transform per scan file) that must be applied to the data read from those files.

Fields

scan_files: FilteredEngineData

Filtered engine data with one row per file to scan (and only selected rows should be scanned)

scan_file_transforms: Vec<Option<ExpressionRef>>

Row-level transformations to apply to data read from files.

Each entry in this vector corresponds to a row in the scan_files data. The entry is an optional expression that must be applied to convert the file’s data into the logical schema expected by the scan:

• Some(expr): Apply this expression to transform the data to match Scan::logical_schema().
• None: No transformation is needed; the data is already in the correct logical form.

Note: This vector can be indexed by row number, as rows masked by the selection vector will have corresponding entries that will be None.

Implementations

impl ScanMetadata

Request that the kernel call a callback on each valid file that needs to be read for the scan.

The arguments to the callback are:

• context: an &mut context argument. this can be anything that engine needs to pass through to each call
• scan_file: a ScanFile struct with all the information about the file

Context

A note on the context. This can be any value the engine wants. This function takes ownership of the passed arg, but then returns it, so the engine can repeatedly call visit_scan_files with the same context.

Example

let mut context = [my context];
for res in scan_metadata_iter { // scan metadata iterator from scan.scan_metadata()
    let scan_metadata = res?;
    context = scan_metadata.visit_scan_files(
       context,
       my_callback,
    )?;
}

pub fn visit_scan_files<T>( &self, context: T, callback: ScanCallback<T>, ) -> DeltaResult<T>

Trait Implementations

impl HasSelectionVector for ScanMetadata

fn has_selected_rows(&self) -> bool

Available on crate feature internal-api only.

Check if the selection vector contains at least one selected row