Struct DatasetSchema 

pub struct DatasetSchema {

    pub schema: Arc<Schema>,
    pub columns: Vec<ColumnDrift>,
    pub omitted: Vec<Vec<(PlSmallStr, DataType)>>,
    pub unreadable: Vec<usize>,
    pub files: usize,
    pub groups: Vec<DriftGroup>,
    pub file_group: Vec<u32>,
    pub origin: SchemaOrigin,
    pub read_as_text: Vec<PlSmallStr>,
    pub empty_files: usize,
    pub median_row_group_bytes: Option<usize>,
    pub median_file_bytes: Option<usize>,
    pub column_ranges: HashMap<PlSmallStr, ColumnRange>,
    pub partition_layouts: Vec<(Vec<String>, usize)>,
    pub partition_layouts_dropped: (usize, usize),
    pub skipped: SkippedFiles,
    pub listed_files: usize,
}

One dataset’s schema, and what deciding it revealed.

Fields

schema: Arc<Schema>

columns: Vec<ColumnDrift>

omitted: Vec<Vec<(PlSmallStr, DataType)>>

Per file, the columns not read from it and the type it holds each in; empty when all fit. The type is the only way back to the values: reading the column as text reads each file at its own written type.

unreadable: Vec<usize>

Files whose footer could not be read, by index into the files given.

files: usize

Files whose footer was read, readable or not.

groups: Vec<DriftGroup>

The distinct ways files differ from the schema; group 0 is “nothing missing”, as an unread file counts.

file_group: Vec<u32>

Per file, in the order given, its group in groups.

origin: SchemaOrigin

read_as_text: Vec<PlSmallStr>

Columns read as text from every file instead of the majority type; empty as the footers found it.

empty_files: usize

Files whose footer said they hold no rows, among those read.

median_row_group_bytes: Option<usize>

The median row group’s compressed size over all footers read; None if none reported one.

median_file_bytes: Option<usize>

The middle file’s size, over the footers read. None when none was read.

column_ranges: HashMap<PlSmallStr, ColumnRange>

Where each column not in every file sits, when its files form a nameable shape. Empty without partitions, or when footers were sampled (an unread file looks complete).

partition_layouts: Vec<(Vec<String>, usize)>

The distinct partition layouts and their file counts, commonest first; usually one or none. From every file’s name, which the listing knows and the footers do not.

partition_layouts_dropped: (usize, usize)

The layouts past those kept, and their files: not remembered in full, but counted.

skipped: SkippedFiles

What the listing passed over on the way to the files read.

listed_files: usize

File names read to find the layouts, including those without partition keys.

Implementations

impl DatasetSchema

pub fn drifting(&self) -> impl Iterator<Item = &ColumnDrift>

Columns that are not in every file, or whose type had to give way.

pub fn with_partition_layouts( self, root: &str, paths: &[String], ) -> DatasetSchema

The dataset with its partition layouts counted from file names, keys taken below root (directories above the opened root are not in dispute). Keys compare as a set: Polars matches hive columns by name, so y=1/m=1 and m=2/y=2 agree.

pub fn with_skipped(self, skipped: SkippedFiles) -> DatasetSchema

Record what the listing passed over. See SkippedFiles.

pub fn reading_as_text(&self, as_text: &[PlSmallStr]) -> DatasetSchema

This dataset as it reads with as_text read as text from every file: for the panel and the table, not the scan (which needs the footer types, so omitted is kept and the caller keeps the original). Those columns stop conflicting; absent stays (a file without the column still shows none).

pub fn drifts(&self) -> bool

Whether any file is missing anything; if not, the scan is plain and rows carry nothing.

Trait Implementations

impl Clone for DatasetSchema

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DatasetSchema

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.