Struct DatasetSchema 

pub struct DatasetSchema {

    pub schema: Arc<Schema>,
    pub columns: Vec<ColumnDrift>,
    pub omitted: Vec<Vec<(PlSmallStr, DataType)>>,
    pub unreadable: Vec<usize>,
    pub files: usize,
    pub groups: Vec<DriftGroup>,
    pub file_group: Vec<u32>,
    pub origin: SchemaOrigin,
    pub read_as_text: Vec<PlSmallStr>,
    pub empty_files: usize,
    pub median_row_group_bytes: Option<usize>,
    pub median_file_bytes: Option<usize>,
    pub column_ranges: HashMap<PlSmallStr, ColumnRange>,
    pub partition_layouts: Vec<(Vec<String>, usize)>,
    pub partition_layouts_dropped: (usize, usize),
    pub skipped: SkippedFiles,
    pub listed_files: usize,
}

One dataset’s schema, and what deciding it revealed.

Fields

schema: Arc<Schema>

columns: Vec<ColumnDrift>

omitted: Vec<Vec<(PlSmallStr, DataType)>>

Per file, in the order given, the columns not read from it and the type that file holds each of them in. Empty for a file whose types all fit.

The type is kept because it is the only way back to the values: reading such a column as text means reading it from each file at the type that file wrote, which the footers know and nothing else does.

unreadable: Vec<usize>

Files whose footer could not be read, by index into the files given.

files: usize

Files whose footer was read, readable or not.

groups: Vec<DriftGroup>

The distinct ways this dataset’s files differ from its schema. Group 0 is always “nothing missing”, which is what a file not read counts as.

file_group: Vec<u32>

Per file, in the order given, its group in groups.

origin: SchemaOrigin

read_as_text: Vec<PlSmallStr>

Columns being read as text from every file rather than as the type most rows have. Empty for a dataset as its footers found it.

empty_files: usize

Files whose footer said they hold no rows, among those read.

median_row_group_bytes: Option<usize>

The middle row group’s compressed size, over every row group of every footer read. None when no footer reported one.

median_file_bytes: Option<usize>

The middle file’s size, over the footers read. None when none was read.

column_ranges: HashMap<PlSmallStr, ColumnRange>

Where each column that is not in every file sits, for those whose files fall into a shape worth naming. Empty for a dataset with no partitions, and for one whose footers were sampled — a file whose footer was not read looks like a file missing nothing, and a range drawn over those would be a guess.

partition_layouts: Vec<(Vec<String>, usize)>

The distinct ways the dataset’s files are partitioned, and how many files are laid out each way, commonest first. One entry, or none, for a dataset whose directories agree — which is nearly all of them.

From the names of every file, not from the footers: this is the one thing the listing knows that reading a file cannot tell you.

partition_layouts_dropped: (usize, usize)

The layouts past the ones kept, and the files under them. A dataset with a key per file is not worth remembering in full, but a note that counts what it does not name has to count all of it.

skipped: SkippedFiles

What the listing passed over on the way to the files it read. From the listing, which had to look at every name anyway.

listed_files: usize

How many file names were read to find the layouts, including the ones with no partition keys at all.

Implementations

impl DatasetSchema

pub fn drifting(&self) -> impl Iterator<Item = &ColumnDrift>

Columns that are not in every file, or whose type had to give way.

pub fn with_partition_layouts( self, root: &str, paths: &[String], ) -> DatasetSchema

The dataset with the ways its files are partitioned counted from their names.

root is the dataset as opened; the keys are taken from below it. A directory above the root is not in dispute — opening run=7/ for a dataset partitioned by date does not make run one of the things its directories disagree about, and counting it made every file look like it disagreed with every other.

Keys are compared as a set: y=1/m=1 and m=2/y=2 partition by the same two things and Polars matches hive columns by name, not position, so a dataset that mixes the orders reads perfectly well and has nothing to disagree about.

pub fn with_skipped(self, skipped: SkippedFiles) -> DatasetSchema

Record what the listing passed over. See SkippedFiles.

pub fn reading_as_text(&self, as_text: &[PlSmallStr]) -> DatasetSchema

This dataset as it reads with as_text read as text from every file.

What the panel shows and what the table draws from, not what the scan is built from — the scan needs the types the footers found, which is why omitted is carried through untouched and why the caller keeps the original alongside.

Those columns stop conflicting: their cells hold a value from every file, so nothing is unread, nothing is left out of a filter or sort, and the note that said the column was not read there has nothing left to say. absent is left alone — a file that never had the column still has none to show.

pub fn drifts(&self) -> bool

Whether any file is missing anything. When nothing is, the scan is one plain read and rows need carry nothing.

Trait Implementations

impl Clone for DatasetSchema

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DatasetSchema

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.