Struct WebDataset 

pub struct WebDataset { /* private fields */ }

A dataset read from WebDataset-format shards.

Build one with WebDataset::builder, then append stages with the methods below. Anything that changes the item type — batching, projecting to tuples — happens on iter using the adapters in filters.

Implementations

impl WebDataset

pub fn builder(urls: impl AsRef<str>) -> WebDatasetBuilder

Start building a dataset from brace patterns or a ::-separated list.

pub fn builder_from<S: AsRef<str>>( patterns: impl IntoIterator<Item = S>, ) -> WebDatasetBuilder

Start building a dataset from several patterns.

pub fn builder_verbatim<S: Into<String>>( urls: impl IntoIterator<Item = S>, ) -> WebDatasetBuilder

Start building a dataset from URLs that need no expansion.

pub fn builder_from_yaml(spec: impl Into<String>) -> WebDatasetBuilder

Available on crate feature yaml only.

Start building a dataset from a YAML multi-source specification.

pub fn open(urls: impl AsRef<str>) -> Result<WebDataset>

Build a dataset from urls with the default settings.

Shards are not shuffled and the dataset refuses to run distributed; use builder to change either.

pub fn from_pipeline(pipeline: DataPipeline) -> WebDataset

Wrap an already assembled pipeline.

pub fn seed(&self) -> u64

The seed the shard shuffle and resampling were built with.

pub fn pipeline(&self) -> &DataPipeline

The underlying pipeline.

pub fn into_pipeline(self) -> DataPipeline

Take the underlying pipeline.

pub fn with(self, stage: impl Stage + 'static) -> WebDataset

Append an arbitrary stage.

pub fn shuffle(self, bufsize: usize) -> WebDataset

Shuffle samples through a buffer of bufsize.

pub fn decode(self, decoder: Decoder) -> WebDataset

Decode every field of every sample.

pub fn decode_basic(self) -> WebDataset

Decode with the default handler chain.

pub fn decode_images(self, imagespec: &str) -> Result<WebDataset>

Available on crate feature image only.

Decode images with the given imagespec, plus the default handlers.

pub fn map( self, f: impl Fn(Sample) -> Result<Option<Sample>> + Send + Sync + 'static, ) -> WebDataset

Apply a function to each sample; returning None drops it.

pub fn map_field( self, field: impl Into<String>, f: impl Fn(Value) -> Result<Value> + Send + Sync + 'static, ) -> WebDataset

Apply a function to one field of each sample.

pub fn select( self, predicate: impl Fn(&Sample) -> bool + Send + Sync + 'static, ) -> WebDataset

Keep the samples a predicate accepts.

pub fn rename<A: Into<String>, B: Into<String>>( self, renames: impl IntoIterator<Item = (A, B)>, ) -> WebDataset

Rename fields, resolving each source from a ;-separated alternation.

pub fn slice(self, start: usize, count: Option<usize>) -> WebDataset

Take count samples starting at start.

pub fn with_epoch(self, nsamples: usize) -> WebDataset

Make an epoch exactly nsamples long, replaying the source as needed.

Each DataLoader worker runs its own copy of the pipeline, so the limit applies per worker: with four workers, with_epoch(1000) yields 4000 samples per epoch. Divide by the worker count if you want the total to match.

pub fn repeat(self, epochs: usize) -> WebDataset

Replay the dataset epochs times.

pub fn take(self, nsamples: usize) -> WebDataset

Stop after nsamples in total.

pub fn iter(&self) -> SampleStream

Start an epoch.

pub fn iter_ok(&self) -> impl Iterator<Item = Sample> + Send

Start an epoch, discarding failures.

pub fn loader(&self) -> DataLoader

Read this dataset with a pool of worker threads.

Trait Implementations

impl Clone for WebDataset

fn clone(&self) -> WebDataset

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WebDataset

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl IntoIterator for &WebDataset

type Item = Result<Sample, Error>

The type of the elements being iterated over.

type IntoIter = Box<dyn Iterator<Item = Result<Sample, Error>> + Send>

Which kind of iterator are we turning this into?

fn into_iter(self) -> SampleStream

Creates an iterator from a value.