Module discover 

Dataset discovery for the home screen.

This is deliberately not a catalogue. Nothing here is persisted: every listing is computed from the filesystem when asked for, and forgotten when the session ends. The only state datui keeps between runs is a list of recently opened paths.

Discovery is also deliberately shallow. Interesting datasets tend to live on mounts — network filesystems, spinning disks, hive trees with a hundred thousand partition files — so a recursive walk would make the home screen slowest exactly where the data is most interesting. Every function here scans one directory level and stops.

Structs

Cost

What pressing Enter on a dataset will actually cost.

Entry

One row on the home screen.

Holds

What one listing of a directory found in it, counted rather than judged.

HowRead

How opening a file row will read it. See how_read.

Partitions

How a hive dataset is laid out on disk.

Scan

What one directory listing produced, and whether it saw all of it.

TableOf

What a row inside a file of tables says about its table.

Enums

DirectoryFormat

What the data files sitting directly in a directory say about how to read it.

EntryKind

What a home-screen row represents.

Sniffed

What a listing finds a file to be by its first bytes.

Constants

CLASSIFIER_VERSION

Bumped whenever a build starts classifying something differently.

MAX_ENTRIES_PER_DIR

Upper bound on entries read from a single directory, so a pathological directory cannot hang the UI.

NO_READER

What the home screen says of a file unreadable_by_name turns away.

Functions

classify_directory

Classify a directory without walking it.

column_names

Column names from a Parquet footer.

data_extension

The extension that says what a file is, with any compression suffix walked past.

data_format

The format a file’s name says it holds, compression suffix walked past.

database_rows

The rows of a file of tables’ listing on the home screen: a database’s tables and views, or an archive’s arrays, by name, as a directory lists its files, SQLite’s own marked to be hidden, each at its path inside the file.

directory_format

What DirectoryFormat the data files directly in dir amount to.

enrich

Fill in row and column counts for a dataset, from Parquet footers only.

enrich_as

As enrich, reading each file the way the open that follows will read it.

enrich_parquet

Fill in row and column counts for a Parquet file from its footer.

enrich_tables

A file of tables’ tables (a SQLite database’s schema, a NumPy archive’s directory): how many of its own, whether Enter opens one of them, and the columns of the one when there is one. A file whose name says a format its bytes must say (a .db file that is not SQLite) is one datui cannot open.

enrich_with

As enrich_as, taking a dataset’s measure from the shape an open kept of it in remembered, where its files are as they were then, rather than from a sample of its footers.

format_age

Render “how long ago” compactly (2d, 3h).

format_rows

Render a row count compactly (2.4M).

format_size

Render a byte count compactly for a listing (340 MB).

has_no_extension

Whether a file’s name has no extension at all: part-00000, LICENSE.

has_parquet_magic

Whether a local file is Parquet by its contents: PAR1 at both ends.

hive_leaf_format

What the files under a hive root’s key=value partitions actually are.

how_read

How opening entry will read it, as crate::FileFormat::read_mode says for its format and how it is stored, and whether a remote one is downloaded first. None for anything but a file whose name says its format.

is_bookkeeping

Whether a listing entry is bookkeeping rather than data.

is_data_file

Whether a path looks like something datui can open.

is_parquet_key

Whether an object key or path is Parquet: named .parquet, or a part file with no extension inside a directory named .parquet, as Spark and GBIF write them (occurrence.parquet/000001). Hidden and job files (_SUCCESS, .crc) are not.

is_parquet_path

Whether a path names a Parquet file: by its extension, or by sitting as a part file with no extension inside a .parquet directory.

is_partition_name

Whether a name is a hive partition (year=2024): key=value, with a non-empty key. The value may be empty in practice.

look_at_directory

The kind and what the listing found, from one read of it.

name_spec_file

Name entry a file of spec, which reads it; a spec that reads its records as several variants makes it a place too, whose tables → lists.

name_unlisted_file

Name a local file row no listing classified (a recent) by the spec that reads it, as a listing names one: a spec’s glob, else, when its name says nothing, the magic in its first bytes.

partition_layout

Describe how a hive dataset is partitioned, from directory names alone.

physical_facts

Pull layout and compression out of a footer that has already been read.

scan_dir

List one directory level, classified. Never recurses.

scan_dir_bounded

List one directory, doing a bounded amount of work regardless of what is in it.

scan_dir_progressive

scan_dir_bounded, handing progress the rows read so far, sorted, every [LISTING_PROGRESS_EVERY] while the read goes on. A directory a share takes seconds to list shows its first rows as they arrive rather than a spinner until the last.

scan_dir_specs

scan_dir_bounded, naming the files it looks inside by formats too: a file whose first bytes carry a spec’s magic is listed as that spec’s.

schema_preview

Read a dataset’s column names and types without reading any data.

sniff_format

What a file with no usable extension turns out to be, from the bytes at its start.

sniff_listed

sniff_format, and when no format datui reads says it, the format spec that reads it as an open would pick one: by glob, else by magic and match.where. One read of the file’s head answers both, so a listing reads nothing more for specs.

split_row

The row of a split named by its path inside its cache directory, as a recent is listed: None when the path names no split of one.

split_rows

The rows of a Hugging Face cache directory’s splits, each at its path inside the directory (cache/test): opened, it is the directory read with --table. Empty for any other directory, and for a cache of one split, which its door opens.

table_row

The row of a table inside a file of tables named by its path (app.db/users), as a recent is listed: None when the path names no table of such a file.

unreadable_by_name

Whether a file’s name already says datui will not open it: an extension no reader takes, under any compression suffix. A bare data.gz is left to the open, which looks inside, and so is a name with no extension.

variant_row

The row of a variant named by its path inside its file (day.itch/add), as a recent is listed: None when the path names no variant of such a file.

variant_rows

The rows of a file a format spec reads as several variants, one a variant, each at its path inside the file (day.itch/add): opened, it is the file read with --table. Empty for any other file.

worth_sniffing

Whether a listing looks inside a file to say what it is: one with no extension, or one whose extension says nothing (.bin, which ArduPilot’s logs and model checkpoints share with everything else) or only text (.log, which candump writes; .txt).

Type Aliases

SchemaPreview

Column name and type, for the home screen’s preview pane.