Crate dataset_core 

A generic, thread-safe dataset container with lazy loading and caching.

dataset-core provides Dataset<T, E>, a lightweight wrapper that pairs a storage directory with a lazily-initialized value of any type T. The actual downloading and parsing logic is supplied by the caller through a loader closure stored at construction time, making Dataset<T, E> suitable for any data source — local files, remote URLs, databases, or in-memory generation.

On top of this core type, the crate offers an optional feature-gated module:

• utils — helper functions for downloading files, extracting archives, verifying SHA-256 hashes, and managing temporary directories.

Ready-to-use loaders for classic ML datasets (Iris, Boston Housing, Diabetes, Titanic, Wine Quality) live in the companion crate dataset-ml, which depends on dataset-core with the utils feature enabled and serves as the reference implementation for wrapping Dataset<T, E>.

Feature Flags

Feature	What it enables
utils	download_to, unzip, acquire_dataset, and the error module

With no features enabled, only Dataset<T, E> is available — depending only on std::sync::OnceLock.

Quick Start — Dataset<T, E>

use dataset_core::Dataset;

fn my_loader(dir: &str) -> Result<Vec<String>, std::io::Error> {
    // In a real use case you would read/download files from `dir`.
    Ok(vec!["hello".to_string(), "world".to_string()])
}

// The loader is supplied once, at construction time.
let mut ds: Dataset<Vec<String>, std::io::Error> = Dataset::new("./my_data", my_loader);

// First call runs the loader; subsequent calls return the cached reference.
let data = ds.load().unwrap();
assert_eq!(data.len(), 2);

let data_again = ds.load().unwrap();
assert!(std::ptr::eq(data, data_again)); // same reference, no reload

// `get` borrows the cached value without ever running the loader;
// `get_mut` edits it in place (no clone, no reload — the change stays cached).
assert!(ds.get().is_some());
if let Some(v) = ds.get_mut() {
    v[0] = "HELLO".to_string();
}
assert_eq!(ds.get().unwrap()[0], "HELLO");

// Move the cached value out without cloning. `take` leaves `ds` reusable
// (a later `load` re-runs the loader); `into_inner` consumes `ds`.
let owned = ds.take().unwrap();
assert_eq!(owned.len(), 2);
assert!(!ds.is_loaded());

ds.load().unwrap(); // `take` reset the cache, so this reloads
let owned = ds.into_inner().unwrap();
assert_eq!(owned.len(), 2);

Swapping the loader

Because the loader lives inside the Dataset, you change how the data is parsed with Dataset::set_loader, which also invalidates the cache so the next access re-parses with the new loader. To re-run the same loader (e.g. the file on disk changed), use Dataset::invalidate.

use dataset_core::Dataset;

let mut ds: Dataset<i32, std::convert::Infallible> = Dataset::new("./data", |_| Ok(1));
assert_eq!(*ds.load().unwrap(), 1);

ds.set_loader(|_| Ok(2)); // swap the loader; old cache is dropped
assert!(!ds.is_loaded());
assert_eq!(*ds.load().unwrap(), 2); // next load uses the new loader

Utility Functions (feature utils)

• download_to — download a remote file into a directory
• unzip — extract a ZIP archive
• acquire_dataset — cache-aware dataset acquisition workflow (temp dir → prepare → optional hash check → move to final location)

acquire_dataset is the single entry point for caching a dataset file; temp-dir creation and SHA-256 verification are internal steps it performs for you.

Re-exports

pub use error::DataFormatErrorKind;

pub use error::DatasetError;

pub use utils::acquire_dataset;

pub use utils::download_to;

pub use utils::unzip;

Modules

error

Error handling module.

utils

Utility functions for dataset authors.

Structs

Dataset

A generic, thread-safe dataset container with lazy loading and in-memory caching.