Crate webdataset_core 

Core data model for the WebDataset format.

WebDataset stores a dataset as a set of POSIX tar archives (“shards”). The files inside a shard that share a basename make up one training sample:

shard-000000.tar
  image0001.jpg   image0001.cls   image0001.json
  image0002.jpg   image0002.cls   image0002.json

This crate defines the pieces every other crate in the workspace shares:

• Sample and Value — the in-memory shape of a training example.
• Tensor and DType — dense numeric arrays, plus the npy codec.
• Error and handlers — how failures are reported and absorbed.
• braceexpand() and utils — shard-list expansion and worker identity.

Most users should depend on the webdataset crate instead, which re-exports everything here.

no_std

Turning off the std feature builds against core and alloc only, which covers the whole data model: samples, values, tensors, the .npy codec, brace expansion, and the error and handler types. What std adds is the parts that need an operating system — Error::Io, environment variable substitution, reading worker identity from the environment, and the stream-based npy helpers.

webdataset-core = { version = "0.1", default-features = false, features = ["json"] }

Threads

The threads feature — on by default — makes per-worker state thread-safe. With the standard library that is a thread-local and needs no setup. Without it, there is no portable way to ask which thread is running, so the host supplies one with workers::set_thread_id_hook; see the workers module. A single-threaded no_std program needs neither.

Re-exports

pub use braceexpand::braceexpand;

pub use braceexpand::braceexpand_all;

pub use error::Error;

pub use error::Result;

pub use error::ResultExt;

pub use fields::FieldHasher;

pub use fields::Fields;

pub use handlers::Action;

pub use handlers::Dispatch;

pub use handlers::Handler;

pub use handlers::HandlerRef;

pub use sample::Sample;

pub use tensor::DType;

pub use tensor::Tensor;

pub use utils::WorkerInfo;

pub use utils::base_plus_ext;

pub use utils::expand_urls;

pub use utils::worker_info;

pub use value::CustomValue;

pub use value::Value;

pub use workers::set_thread_id_hook;

pub use workers::set_worker;

pub use workers::with_worker;

Modules

braceexpand

Bash-style brace expansion.

error

Error type shared across the webdataset crates.

fields

The map type used for sample fields.

handlers

Pluggable exception handlers.

npy

Reading and writing NumPy’s .npy and .npz container formats.

sample

The Sample type: one training example, assembled from the files in a tar archive that share a basename.

tensor

A minimal dense n-dimensional array.

utils

Shared helpers: filename splitting, worker identity, seeds, and secure mode.

value

The dynamically typed values that make up a Sample.

workers

Which worker the calling thread is, and how that is discovered.