Module hf_splits 

The splits of a Hugging Face datasets directory.

A cache directory holds a file per split, name-train.arrow, or a split’s shards, name-train-00000-of-00003.arrow, beside dataset_info.json. A save_to_disk DatasetDict holds a subdirectory per split, named in its dataset_dict.json. The splits are separate tables: an open reads one, train unless --table names another, and lists the rest. map() writes its results beside a cache’s splits as cache-*.arrow, with columns of their own, so those are left out and counted.

Structs

Splits

What an open of a cache directory chose, and what it left out.

Constants

DATASET_DICT

The file that marks a DatasetDict directory.

Functions

cache_splits

The splits a cache directory’s Arrow files name, in the order an open offers them, as the home screen lists them inside it. Empty for any other directory, and for one whose files name no splits. Its listing is all that is read.

choose

Which of names, a cache directory’s Arrow files, an open reads: the files of the split pick chooses, the others by name after the three datasets names itself. Indices into names, in its order.

dataset_dict

The splits a save_to_disk DatasetDict directory names in its dataset_dict.json, in its order, each a subdirectory of dir. None for any other directory.

dict_splits

The split names of a dataset_dict.json: {"splits": ["train", "test"]}. Each is a directory’s name, so one that could name another place is refused.

is_cache

Whether name is a file map() wrote rather than a split’s.

is_cache_dir

Whether dir is a datasets cache directory: its metadata beside Arrow files.

pick

The split of listed an open reads, table if given, else the first in offered order, and the others in that order: train, validation and test, then the rest as listed.

split_of

The split a cache file’s name says it holds: train for people-train.arrow and people-train-00001-of-00002.arrow. Split names are word characters and dots.

split_place

The cache directory and the split a path inside one names (cache/test), as the home screen lists it and recents record it. None for a path that is there.