Struct DynamicScan 

pub struct DynamicScan {
    pub schema: SchemaRef,
    pub file_type: FileType,
    pub base_url: Url,
    pub file_constant_columns: Vec<String>,
    pub path_column: ColumnName,
    pub file_size_column: ColumnName,
    pub last_modified_column: ColumnName,
    pub dv_column: ColumnName,
}

Available on crate feature declarative-plans only.

Reads data files from an upstream stream of file-metadata tuples, one input row per file. For each row, the path, size, and last-modified columns describe the file; the engine resolves its path against base_url (see below), opens it as file_type, and reads columns matching schema.

file_constant_columns lists upstream columns whose per-file values are broadcast onto every emitted file row. This is file-constant metadata, the same concept as ScanParquet::file_constant_columns. Each named input field must have the same type and nullability as its output field. See the example below.

dv_column names a nullable column on the upstream row holding a Delta DeletionVectorDescriptor struct. The engine resolves it into a roaring bitmap and drops file rows whose row index appears in the DV. A NULL value for a given input row means “no DV for this file”, so all file rows are emitted.

Each path value is resolved against base_url via Url::join. URL-reference resolution need not stay under base_url: a different-scheme absolute URL replaces the base, while a value starting with / or // replaces its path or authority.

Output row order is unspecified: the engine is free to read files in any order, in parallel, and to interleave rows from different files. The relative order of upstream rows is not preserved.

Example

Given an upstream metadata stream and a DynamicScan configuration:

upstream (metadata)
    path             | size | filemod | version | dv
    -----------------+------+---------+---------+------
    part-0.parquet   | 1024 |  100000 |       7 | NULL
    part-1.parquet   | 2048 |  200000 |       8 | NULL

DynamicScan {
    schema: { id: int, name: string, version: long },
    file_type: Parquet,
    base_url: "s3://table/",
    file_constant_columns: ["version"],
    path_column: "path",
    file_size_column: "size",
    last_modified_column: "filemod",
    dv_column: "dv",
}

The engine opens s3://table/part-0.parquet and s3://table/part-1.parquet, reads {id, name} from each, sees a NULL DV for each file so all rows survive, and broadcasts the row’s version onto every emitted file row. One possible output (row order is not guaranteed):

    | id | name | version
    +----+------+--------
    |  3 |  c   |       8
    |  2 |  b   |       7
    |  4 |  d   |       8
    |  1 |  a   |       7

Fields

schema: SchemaRef

file_type: FileType

base_url: Url

Hierarchical base URL ending in / against which per-row path values resolve.

file_constant_columns: Vec<String>

path_column: ColumnName

Non-nullable input column holding the per-row file path or URL fragment.

file_size_column: ColumnName

Non-nullable input column with the file’s total size in bytes.

last_modified_column: ColumnName

Non-nullable input column with the last-modified timestamp in milliseconds since epoch.

dv_column: ColumnName

Nullable input column with the schema of DeletionVectorDescriptor.

Implementations

impl DynamicScan

pub fn try_new( input_schema: &SchemaRef, output_schema: impl Into<SchemaRef>, file_type: FileType, base_url: Url, file_constant_columns: impl IntoIterator<Item = impl Into<String>>, path_column: ColumnName, file_size_column: ColumnName, last_modified_column: ColumnName, dv_column: ColumnName, ) -> DeltaResult<Self>

Constructs a DynamicScan whose emitted rows match output_schema.

input_schema describes the upstream rows containing file metadata. The scan reads file_type files relative to base_url.

Errors

Returns an error when base_url is not hierarchical or does not end in /; when a required metadata or deletion-vector column is absent from input_schema, has an incompatible type, or has invalid nullability; or when a file-constant column is absent from either schema, is a metadata column, or has different input and output types or nullability.

pub fn validate_input(&self, input_schema: &SchemaRef) -> DeltaResult<()>

Validates the columns consumed by this scan against an upstream input_schema.

Returns Ok(()) when the base URL is valid and every configured column resolves with the required type and nullability.

Errors

Returns an error when base_url is not hierarchical or does not end in /; when a required metadata or deletion-vector column is absent, has an incompatible type, or has invalid nullability; or when a file-constant column is absent from either schema, is a metadata column, or has different input and output types or nullability.

Trait Implementations

impl Clone for DynamicScan

fn clone(&self) -> DynamicScan

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DynamicScan

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&DynamicScan> for DynamicScanNode

fn from(node: &DynamicScan) -> Self

Converts to this type from the input type.

impl From<DynamicScan> for Operator

fn from(payload: DynamicScan) -> Self

Converts to this type from the input type.