Enum DirectoryFormat 

pub enum DirectoryFormat {
    One(FileFormat, Vec<PathBuf>),
    Mixed {
        format: FileFormat,
        files: Vec<PathBuf>,
        passed_over: Vec<(FileFormat, usize)>,
    },
    Deeper,
}

What the data files sitting directly in a directory say about how to read it.

A directory opened as one dataset has to be read by something, and the only honest source for that is the files in it. Before this existed the answer was assumed: any directory was scanned as Parquet, so a directory of .json.gz was opened by seeking to the end of each file for a PAR1 that was never going to be there.

Variants

One(FileFormat, Vec<PathBuf>)

Every data file directly in the directory reads as this one format, and these are the files. Sorted, because a concatenation’s row order is its file order.

Mixed

The files name more than one format. The commonest of them is the table — a directory of a thousand CSVs and one stray JSON is a directory of CSVs — and the rest are counted by format so the read can say what it passed over. Parquet wins a tie, because it is the format a directory of data files is most likely to be about and the one every other route here reads in place.

Fields

format: FileFormat

files: Vec<PathBuf>

passed_over: Vec<(FileFormat, usize)>

Deeper

The directory settles nothing by itself: it holds no readable data file directly, or it has subdirectories and so may hold its data below. A hive dataset looks like this — its files are a level down, under key=value.

Trait Implementations

impl Clone for DirectoryFormat

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DirectoryFormat

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for DirectoryFormat

impl PartialEq for DirectoryFormat

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for DirectoryFormat