Struct Dataset 

pub struct Dataset<T, E> { /* private fields */ }

A generic, thread-safe dataset container with lazy loading and in-memory caching.

Dataset<T, E> is a thin caching wrapper that holds a storage_dir (the directory where dataset files are stored on disk), a loader closure, and a lazily-initialized value of type T. The caller supplies the download and parse logic through the loader passed to Dataset::new. Dataset::load runs this loader on first access.

This struct serves as the building block for the loaders in the companion crate dataset-ml and for custom datasets that external users define.

Type Parameters

• T - The type of the parsed dataset. It can be any type, such as (Array2<f64>, Array1<f64>), a custom struct, or another data shape. T must implement Send + Sync so that Dataset<T, E> can be shared across threads.
• E - The error type returned by the loader. Callers choose it freely, for example std::io::Error, a crate-specific DatasetError, or std::convert::Infallible for loaders that cannot fail.

Thread Safety

Dataset<T, E> is Send + Sync when T is Send + Sync (the stored loader is always Send + Sync). The loader runs at most once even when multiple threads call Dataset::load concurrently. An internal mutex serializes the first load, so late arrivals wait for it and then share its result, rather than each thread starting its own download.

Example

use dataset_core::Dataset;

// Define a simple loader that reads a value from the storage directory path.
// The loader can return any error type you choose.
fn my_loader(dir: &str) -> Result<Vec<String>, std::io::Error> {
    // A real use case downloads or reads files from `dir`.
    // This shows the caching behavior.
    Ok(vec!["hello".to_string(), "world".to_string()])
}

// The loader is bound to the dataset at construction time.
let mut dataset: Dataset<Vec<String>, std::io::Error> = Dataset::new("./my_data", my_loader);

// The first call to `load` triggers the loader
let data = dataset.load().unwrap();
assert_eq!(data.len(), 2);

// Subsequent calls return the cached reference instantly
let data_again = dataset.load().unwrap();
assert!(std::ptr::eq(data, data_again)); // same reference, no re-load

// Check whether data has been loaded
assert!(dataset.is_loaded());

// Borrow the cached value without reloading, or edit it in place with `get_mut`.
if let Some(v) = dataset.get_mut() {
    v[0] = "HELLO".to_string();
}
assert_eq!(dataset.get().unwrap()[0], "HELLO");

// Move the cached value out without cloning.
// `take` leaves `dataset` reusable. `into_inner` consumes it.
let owned = dataset.take().unwrap();
assert_eq!(owned.len(), 2);
assert!(!dataset.is_loaded()); // `take` resets it to unloaded

dataset.load().unwrap(); // this reloads, because `take` cleared the cache
let owned = dataset.into_inner().unwrap();
assert_eq!(owned.len(), 2);

Implementations

impl<T, E> Dataset<T, E>

pub fn new( storage_dir: &str, loader: impl Fn(&str) -> Result<T, E> + Send + Sync + 'static, ) -> Self

Create a new Dataset instance without loading any data.

This is a lightweight operation that only stores the storage directory path and the loader. It performs no I/O or network requests until Dataset::load runs.

Parameters

• storage_dir - The directory where the loader stores dataset files. If the directory does not yet exist, the loader creates it automatically when it runs.
• loader - A closure or function that takes the storage directory path (&str) and returns Result<T, E>. This is where you download data, handle file I/O, and parse it. It runs at most once (see Dataset::load). Dataset::new stores it behind Box<dyn Fn ...>, so it must be Send + Sync + 'static. Capture owned values or clones instead of borrowing from the environment.

Returns

A new Dataset<T, E> instance ready for lazy loading.

pub fn load(&self) -> Result<&T, E>

Load the dataset. The first call runs the stored loader and caches the result.

On the first call, load runs the loader supplied to Dataset::new (or last set via Dataset::set_loader) with the storage directory path. It caches the returned value. Later calls, from any thread, return a reference to the cached value without running the loader again.

Concurrency

The loader runs at most once, even when several threads call load simultaneously. Threads that arrive while a load is in flight block until it finishes, then share its result. This matters for the typical loader, which downloads into storage_dir: concurrent callers would otherwise each start their own download of the same file.

A loader that returns Err leaves the Dataset unloaded, so a later load retries it. load does not cache the error.

Returns

• Ok(&T) - A reference to the cached dataset.

Errors

Returns any error the loader produces on the first call. After the first successful load, this method never returns an error.

pub fn load_mut(&mut self) -> Result<&mut T, E>

Load the dataset if needed, then return a mutable reference to it.

This is the loading counterpart of Dataset::get_mut. Unlike get_mut, which returns None when nothing is cached yet, load_mut runs the loader first. It always returns a mutable reference on success. Use it to load the data and adjust it in one step. For example, normalize features right after parsing, rather than calling Dataset::load and Dataset::get_mut separately.

As with get_mut, you edit the value in place, and the change persists in the cache.

Returns

• Ok(&mut T) - A mutable reference to the cached dataset.

Errors

Returns any error the loader produces on the first call.

Example

use dataset_core::Dataset;

let mut ds: Dataset<Vec<i32>, std::convert::Infallible> =
    Dataset::new("./data", |_| Ok(vec![1, 2, 3]));

// Loads on first call, then returns a mutable reference.
ds.load_mut().unwrap().push(4);
assert_eq!(ds.get(), Some(&vec![1, 2, 3, 4])); // the change persisted

pub fn set_loader( &mut self, loader: impl Fn(&str) -> Result<T, E> + Send + Sync + 'static, )

Replace the loader and invalidate any cached data.

Use this when the parsing logic itself needs to change. set_loader does not run the new loader right away. It only swaps the loader and drops the cached value, which resets the Dataset to its unloaded state. The next Dataset::load call then re-parses the data with the new loader. This keeps the “no I/O until access” contract intact.

To re-run the same loader instead, use Dataset::invalidate.

Parameters

• loader - The replacement loader. Like the one given to Dataset::new, it must be Send + Sync + 'static.

Example

use dataset_core::Dataset;

let mut ds: Dataset<i32, std::convert::Infallible> = Dataset::new("./data", |_| Ok(1));
assert_eq!(*ds.load().unwrap(), 1);

ds.set_loader(|_| Ok(2)); // swaps the loader and drops the old cache
assert!(!ds.is_loaded());
assert_eq!(*ds.load().unwrap(), 2); // next load uses the new loader

pub fn invalidate(&mut self)

Drop the cached value, keeping the current loader.

This resets the Dataset to its unloaded state, so the next Dataset::load call re-runs the current loader from scratch. If the underlying files change on disk and you want to re-parse them, call this method. To swap in a different loader, use Dataset::set_loader.

Unlike Dataset::take, this does not return the cached value. It simply discards it.

Example

use dataset_core::Dataset;

let mut ds: Dataset<i32, std::convert::Infallible> = Dataset::new("./data", |_| Ok(1));
ds.load().unwrap();
assert!(ds.is_loaded());

ds.invalidate(); // drop the cache, keep the loader
assert!(!ds.is_loaded());
assert_eq!(*ds.load().unwrap(), 1); // reloads with the same loader

pub fn is_loaded(&self) -> bool

Check whether the dataset has been loaded into memory.

Returns

true if Dataset::load has been called successfully at least once, false otherwise.

pub fn storage_dir(&self) -> &str

Get the storage directory path.

Returns

The storage directory path as a string slice.

pub fn get(&self) -> Option<&T>

Get a reference to the cached value without triggering loading.

Unlike Dataset::load, this never runs the loader. If the dataset is not loaded yet, it returns None instead of downloading or parsing anything. When you want data only if it is already in memory, use get. This avoids the loader’s I/O cost when the data is not cached. For example, a fast path can fall back to other work when the dataset is not yet cached.

This is the reference-returning companion of Dataset::is_loaded: is_loaded() answers whether the value is cached, and get() returns the cached reference when it is.

Returns

• Some(&T) - a reference to the cached value, if the dataset is loaded.
• None - if the dataset is not loaded.

Example

use dataset_core::Dataset;

let ds: Dataset<Vec<i32>, std::convert::Infallible> =
    Dataset::new("./data", |_| Ok(vec![1, 2, 3]));
assert!(ds.get().is_none()); // not loaded yet, no loader runs

ds.load().unwrap();
assert_eq!(ds.get(), Some(&vec![1, 2, 3]));

pub fn get_mut(&mut self) -> Option<&mut T>

Get a mutable reference to the cached value for in-place editing.

This is the only way to mutate the cached value without moving it out. You can tweak the loaded data, for example to normalize features, add missing values, or augment samples. The changes persist in the cache, so later Dataset::load and Dataset::get calls observe them.

Because it needs unique access (&mut self), there is no risk of aliasing or a race. Unlike both take and into_inner, it neither clones nor removes the value. The Dataset stays loaded.

Like Dataset::get, this does not trigger loading. It returns None if the dataset is not loaded. If you need the value to be present, call Dataset::load first.

Returns

• Some(&mut T) - a mutable reference to the cached value, if the dataset is loaded.
• None - if the dataset is not loaded.

Example

use dataset_core::Dataset;

let mut ds: Dataset<Vec<i32>, std::convert::Infallible> =
    Dataset::new("./data", |_| Ok(vec![1, 2, 3]));
assert!(ds.get_mut().is_none()); // not loaded yet, no loader runs

ds.load().unwrap();
if let Some(data) = ds.get_mut() {
    data.push(4); // edit the cached value in place, no clone, no reload
}
assert_eq!(ds.get(), Some(&vec![1, 2, 3, 4])); // the change persisted

pub fn into_inner(self) -> Option<T>

Consume the Dataset and return the cached value, if any.

This moves the cached T out of the container. There is no clone. Because it takes self by value, this consumes the Dataset. You cannot use it afterward.

This method does not trigger loading. It returns None if the dataset was never loaded. If you need the value to be present, call Dataset::load first.

into_inner vs take

Both move the cached value out without cloning. The difference is what happens to the container:

• into_inner takes self and consumes the Dataset. Use it when you are done with the container.
• take takes &mut self, leaving the Dataset reusable in its unloaded state (a later load re-runs the loader).

Returns

• Some(T) - the cached value, if the dataset is loaded.
• None - if the dataset was never loaded.

Example

use dataset_core::Dataset;

let ds: Dataset<Vec<i32>, std::convert::Infallible> =
    Dataset::new("./data", |_| Ok(vec![1, 2, 3]));
ds.load().unwrap();

let owned: Vec<i32> = ds.into_inner().unwrap();
assert_eq!(owned, vec![1, 2, 3]);
// `into_inner` consumed `ds`. You can no longer use it.

// A dataset that was never loaded yields `None`.
let empty: Dataset<Vec<i32>, std::convert::Infallible> =
    Dataset::new("./data", |_| Ok(vec![1, 2, 3]));
assert!(empty.into_inner().is_none());

pub fn take(&mut self) -> Option<T>

Take the cached value out of the Dataset, leaving it reusable.

This moves the cached T out. There is no clone. It also resets the Dataset to its unloaded state. Unlike into_inner, take leaves the container intact. You can use it again, and a later Dataset::load call runs the loader from scratch.

This method does not trigger loading. It returns None if the dataset is not loaded.

take vs into_inner

Both move the cached value out without cloning. The difference is what happens to the container:

• take takes &mut self and keeps the Dataset reusable (reset to unloaded) after extracting the value.
• into_inner takes self and consumes the container entirely.

Returns

• Some(T) - the cached value, if the dataset is loaded.
• None - if the dataset is not loaded.

Example

use dataset_core::Dataset;

let mut ds: Dataset<i32, std::convert::Infallible> = Dataset::new("./data", |_| Ok(1));
ds.load().unwrap();
assert!(ds.is_loaded());

let taken = ds.take().unwrap();
assert_eq!(taken, 1);
assert!(!ds.is_loaded()); // reset to unloaded, but `ds` is still usable

// Because it was reset, `load` runs the loader again:
let reloaded = ds.load().unwrap();
assert_eq!(*reloaded, 1);

Trait Implementations

impl<T, E> Debug for Dataset<T, E>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.