Crate dataset_core 

A generic, thread-safe dataset container with lazy loading and caching.

dataset-core provides Dataset<T, E>, a lightweight wrapper that pairs a storage directory with a lazily-initialized value of any type T. The caller supplies the download and parse logic through a loader closure stored at construction time. This makes Dataset<T, E> suitable for any data source: local files, remote URLs, databases, or in-memory generation.

On top of this core type, the crate offers an optional feature-gated module:

• utils: helper functions to download files, extract archives, verify SHA-256 hashes, and manage temporary directories.

Ready-to-use loaders for classic ML datasets live in the companion crate dataset-ml. Examples include Iris, Breast Cancer, Titanic, Forest CoverType, KDD Cup ’99, and 20 Newsgroups. The crate depends on dataset-core with the utils feature enabled and serves as the reference implementation that wraps Dataset<T, E>.

Feature Flags

Feature	What it enables
utils	download_to, download_to_with_retries, unzip, gunzip, untar, untar_gz, sha256_file, verify_sha256, read_latin1, acquire_dataset, and the error module

With no features enabled, only Dataset<T, E> is available. It depends only on std::sync::OnceLock.

Quick Start: Dataset<T, E>

use dataset_core::Dataset;

fn my_loader(dir: &str) -> Result<Vec<String>, std::io::Error> {
    // A real use case downloads or reads files from `dir`.
    Ok(vec!["hello".to_string(), "world".to_string()])
}

// The caller supplies the loader once, at construction time.
let mut ds: Dataset<Vec<String>, std::io::Error> = Dataset::new("./my_data", my_loader);

// The first call runs the loader. Later calls return the cached reference.
let data = ds.load().unwrap();
assert_eq!(data.len(), 2);

let data_again = ds.load().unwrap();
assert!(std::ptr::eq(data, data_again)); // same reference, no reload

// `get` borrows the cached value. It does not run the loader.
// `get_mut` edits the value in place, with no clone or reload. The change stays cached.
assert!(ds.get().is_some());
if let Some(v) = ds.get_mut() {
    v[0] = "HELLO".to_string();
}
assert_eq!(ds.get().unwrap()[0], "HELLO");

// Move the cached value out without cloning. `take` leaves `ds` reusable.
// A later `load` call re-runs the loader. `into_inner` consumes `ds`.
let owned = ds.take().unwrap();
assert_eq!(owned.len(), 2);
assert!(!ds.is_loaded());

ds.load().unwrap(); // `take` reset the cache, so this reloads
let owned = ds.into_inner().unwrap();
assert_eq!(owned.len(), 2);

Swapping the loader

Because the loader lives inside the Dataset, Dataset::set_loader lets you change how the loader parses the data. It also invalidates the cache, so the next access re-parses with the new loader. If the file on disk changes, use Dataset::invalidate to re-run the same loader.

use dataset_core::Dataset;

let mut ds: Dataset<i32, std::convert::Infallible> = Dataset::new("./data", |_| Ok(1));
assert_eq!(*ds.load().unwrap(), 1);

ds.set_loader(|_| Ok(2)); // swap the loader and drop the old cache
assert!(!ds.is_loaded());
assert_eq!(*ds.load().unwrap(), 2); // next load uses the new loader

Utility Functions (feature utils)

• download_to - download a remote file into a directory
• download_to_with_retries - same as download_to, but retries transient failures with backoff
• unzip - extract a ZIP archive
• gunzip - decompress a gzip (.gz) file into a single output file
• untar - extract a tar (.tar) archive into a directory
• untar_gz - extract a gzip-compressed tar (.tar.gz / .tgz) archive as a stream
• sha256_file - compute a file’s SHA-256 digest, to pin as an expected hash
• verify_sha256 - check a file against a hash you already have
• read_latin1 - read a file as Latin-1 text, with no data loss and no failure on non-UTF-8 bytes
• acquire_dataset - cache-aware dataset acquisition workflow (temp dir → prepare → optional hash check → move to final location)

acquire_dataset is the single entry point for caching a dataset file. It creates a temporary directory and verifies the SHA-256 hash internally. Use sha256_file and verify_sha256 only outside that workflow. Use them to pin a new dataset’s hash, or to check which file is on disk after a test runs.

Re-exports

pub use error::DataFormatErrorKind;

pub use error::DatasetError;

pub use utils::acquire_dataset;

pub use utils::download_to;

pub use utils::download_to_with_retries;

pub use utils::gunzip;

pub use utils::read_latin1;

pub use utils::sha256_file;

pub use utils::untar;

pub use utils::untar_gz;

pub use utils::unzip;

pub use utils::verify_sha256;

Modules

error

Error handling module.

utils

Utility functions for dataset authors.

Structs

Dataset

A generic, thread-safe dataset container with lazy loading and in-memory caching.