Struct DatasetShape 

pub struct DatasetShape {
    pub fingerprint: String,
    pub files: Vec<CachedFooter>,
    pub schemas: Vec<Vec<(String, DataType)>>,
    pub taken_at: u64,
}

What an open learned about a dataset’s files, kept so the next one can show its columns and its row count without reading a single footer.

Reading the footers is what opening a large dataset costs: one read per file, and a prefix of a few thousand objects spends seconds there every time it is opened. The listing is one request and has to happen anyway — it is how datui knows what the dataset is now — so it is the listing that decides whether this is still true, and the footers that it saves.

A cache, not a catalogue, in the same sense as DatasetFacts: every field is re-derivable by reading the dataset again, a fingerprint that no longer matches is ignored, and deleting the file costs speed and nothing else.

Fields

fingerprint: String

What the dataset’s files looked like when this was taken. A listing that comes back with a different one describes a dataset that has changed, and everything below it is then about a dataset that no longer exists.

files: Vec<CachedFooter>

One entry per file, in the order the listing returned them, which is the order a scan reads them in.

schemas: Vec<Vec<(String, DataType)>>

The distinct schemas the files have, as name and type in order.

Held apart and referred to by index because a dataset of ten thousand files usually has one schema, sometimes three, and never ten thousand. Writing each file’s columns out in full would make the cache larger than the footers it saves reading.

The types are Polars’ own, serialised as Polars serialises them, rather than their printed names parsed back. A name is not enough to rebuild a type — a nested or parametrised one prints as something no parser here could take apart again — and a type rebuilt slightly wrong would seat the wrong schema under a dataset that reads fine. Should that representation change under a Polars upgrade, the entries stop parsing and the cache is simply empty, which is the failure this is allowed to have.

taken_at: u64

Seconds since the Unix epoch, for a human reading the file.

Implementations

impl DatasetShape

pub fn fingerprint_of<'a>( files: impl IntoIterator<Item = (&'a str, u64, u64, Option<&'a str>)>, ) -> String

The fingerprint of a listing: which files, how many, how much they weigh, when each was last written, and the store’s own tag for each where it gave one.

Everything a listing can see without opening anything, which is the point — this has to be cheap enough to be worth taking, and all of it arrives in the one response that had to happen anyway. A file added, removed, renamed, resized or rewritten changes it; a dataset that has not been touched does not.

The names are in it because the cache is a positional join: the remembered footers are lined up against a freshly listed dataset by position, so the identity of what is being joined belongs in the thing that says the join is still valid. The ETag is in it because size and a whole-second timestamp cannot see a file overwritten within the same second at the same length.

pub fn intern_schema( schemas: &mut Vec<Vec<(String, DataType)>>, schema: &Schema, ) -> usize

Where schema sits in schemas, added on the end if it is not there yet.

pub fn schema_at( schemas: &[Vec<(String, DataType)>], at: usize, ) -> Option<Schema>

The schema at at in schemas, or None when the table has no such entry.

Trait Implementations

impl Clone for DatasetShape

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DatasetShape

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DatasetShape

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for DatasetShape

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for DatasetShape

impl PartialEq for DatasetShape

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for DatasetShape

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for DatasetShape