Module discover 

Dataset discovery for the home screen. Not a catalog: listings are computed from the filesystem when asked and forgotten at session end; between runs datui keeps only recent paths and measured shapes (remembered, valid while the files are unchanged). Every function scans one directory level, since data lives on slow mounts and huge partition trees.

Structs

Cost

What pressing Enter on a dataset will cost (200 MB of zstd Parquet is gigabytes in memory, and NFS is not tmpfs), all from what datui already reads: the mount table and the footer.

Entry

One row on the home screen.

Holds

What one listing of a directory found, counted rather than judged. The row’s label comes from here (12 parquet), true whether or not the files are one table.

HowRead

How opening a file row will read it. See how_read.

Partitions

How a hive dataset is laid out, from directory names alone.

Rules

Where the local and the bucket listings deliberately differ.

Scan

What one directory listing produced, and whether it saw all of it.

Seen

One entry of a listing, on disk or in a bucket, as classify asks about it.

TableOf

What a row inside a file of tables says about its table.

Enums

DirectoryFormat

What the data files sitting directly in a directory say about how to read it.

EntryKind

What a home-screen row represents.

Sniffed

What a listing finds a file to be by its first bytes.

Constants

CLASSIFIER_VERSION

Bump whenever classification changes. A cached kind is restored without re-deriving it (a remote row cannot be read cheaply), so it is restored only when written by a build with the same version; measurements in the record (rows, columns, cost) survive regardless.

MAX_ENTRIES_PER_DIR

Upper bound on entries read from one directory, so a huge one cannot hang the UI.

NO_READER

What the home screen says of a file unreadable_by_name turns away.

Functions

classify

A directory’s kind and holdings from one listing level: the one rule for look_at_directory and bucket prefixes, so a directory and its mirror agree; Rules holds their differences. The caller sets truncation.

classify_directory

Classify a directory without walking it: one listing bounded by MAX_ENTRIES_PER_DIR (not a probe of the first few entries, whose answer would depend on read order). Includes rows on network mounts: one getdents walk, stat’ing only symlinks. home_open_selected calls it on the key thread; others on workers.

column_names

Column names from a Parquet footer.

data_extension

The extension saying what a file is, past any compression suffix (sales.csv.gz is CSV); None when not something datui reads.

data_format

The format a file’s name says it holds, past any compression suffix. Extensions are not formats (.ipc, .arrow, .arrows, .feather are one); asking crate::FileFormat keeps home from listing what the reader cannot open.

database_rows

A file of tables listed as rows (a database’s tables and views, an archive’s arrays), each at its path inside the file; SQLite’s own marked hidden.

directory_format

What DirectoryFormat the data files directly in dir amount to.

enrich

Fill in row and column counts from Parquet footers: one file, or a bounded sum for hive and multi-file datasets. Non-Parquet keeps None, a blank in the UI.

enrich_as

As enrich, reading files as the following open will: where the header is decides what the names are, so judging with other settings would misjudge (e.g. --no-header).

enrich_parquet

Fill in a Parquet file’s counts from its footer, reading no column data. Other formats keep None (a CSV’s rows need a scan).

enrich_tables

A file of tables’ tables (SQLite schema, NumPy archive directory): how many, whether Enter opens one, and that one’s columns. A file whose bytes contradict its name (a .db that is not SQLite) is unopenable.

enrich_with

As enrich_as, using the shape an open kept in remembered while the files are unchanged, instead of sampling footers.

format_age

Render “how long ago” compactly (2d, 3h).

format_rows

Render a row count compactly (2.4M).

has_no_extension

Whether a file’s name has no extension at all: part-00000, LICENSE.

has_parquet_magic

Whether a local file is Parquet by its contents: PAR1 at both ends.

hive_leaf_format

What the files under a hive root’s partitions are. directory_format can only say Deeper of a root, so this follows one spine (the first partition at each level, as a hive scan reads its schema) to the bottom. A sample: disagreeing partitions report as the first one.

how_read

How opening entry reads it (crate::FileFormat::read_mode for its format and storage), and whether a remote one is downloaded first. None unless a file’s name says its format.

is_bookkeeping

Whether a listing entry is bookkeeping: a leading _ or . (_SUCCESS, _committed_*, _metadata.json, .crc) or the _$folder$ marker. One predicate for every listing and open, so they agree.

is_data_file

Whether a path looks openable by name or place (a part file in a .parquet directory). Every route asks here so they agree.

is_empty_marker

An empty extensionless object: a tool’s folder marker (yellow/year=2032), whether or not the folder still exists. Nothing datui opens is both empty and nameless.

is_parquet_key

Whether a key or path is Parquet: named .parquet, or an extensionless part file in a .parquet directory (Spark, GBIF: occurrence.parquet/000001). Hidden and job files (_SUCCESS, .crc) are not.

is_parquet_path

Whether a path names a Parquet file, by extension or as an extensionless part file in a .parquet directory. Unlike is_parquet_key, a writer’s own file (_manifest.parquet) counts: it can be opened, even if it does not make its directory a dataset.

is_partition_name

Whether a name is a hive partition (year=2024): key=value, with a non-empty key. The value may be empty in practice.

look_at_directory

The kind (what Enter does) and what the listing found (what the label says), from one read.

name_spec_file

Name entry a file of spec; a spec reading several record variants also makes it a place whose tables → lists.

name_unlisted_file

Name an unlisted local file row (a recent) by the spec that reads it, as a listing would: by glob, else by magic when its name says nothing.

partition_layout

Describe how a hive dataset is partitioned, from directory names alone.

physical_facts

Pull layout and compression from an already-read footer: what reading the file will do, beyond its size.

scan_dir_progressive

List one directory with bounded work: one read_dir and a stat per entry, up to MAX_ENTRIES_PER_DIR; errors give an empty listing. Nothing is classified: subdirectories return EntryKind::Unknown and are looked into later from the viewport, so a label is a fact about the row, not its position. progress gets the rows read since the last call, every LISTING_PROGRESS_EVERY, so a slow share shows rows as they arrive.

scan_dir_specs

scan_dir_progressive, also naming sniffed files by formats’ magic.

schema_preview

A dataset’s column names and types without reading data; Parquet only, None otherwise (the UI says so).

sniff_format

What a file with no usable extension is, from its first bytes (Parquet, Arrow, Avro and ORC carry signatures; CSV and JSON have none and are not guessed). Asked only of a directory being opened, never one being looked at; which signatures a listing trusts is each format’s call (crate::formats::readers::Trusted::listing).

sniff_listed

sniff_format, else the format spec an open would pick (by glob, else by magic and match.where), from one read of the file’s head.

split_row

The row of a split named by its path inside its cache directory (a recent); None if none.

split_rows

A Hugging Face cache directory’s splits as rows at their paths inside it (cache/test), opened with --table. Empty otherwise, or for one split (its door opens it).

table_row

The row of a table named by its path inside its file (app.db/users, a recent); None if none.

unreadable_by_name

Whether a name already rules the file out: an extension no reader takes, past any compression suffix. A bare data.gz or extensionless name is left to the open.

variant_row

The row of a variant named by its path inside its file (a recent); None if none.

variant_rows

A spec-read file’s variants as rows at their paths inside it (day.itch/add), opened with --table. Empty for other files.

worth_sniffing

Whether a listing sniffs a file: no extension, an uninformative one (.bin), or only text (.log, which candump writes; .txt).

Type Aliases

SchemaPreview

Column name and type, for the home screen’s preview pane.