Struct HfDataset 

pub struct HfDataset { /* private fields */ }

HuggingFace Hub dataset configuration and loader.

This struct provides a builder pattern for configuring and downloading datasets from the HuggingFace Hub.

Implementations

impl HfDataset

pub fn builder(repo_id: impl Into<String>) -> HfDatasetBuilder

Creates a new builder for a HuggingFace dataset.

Arguments

• repo_id - The dataset repository ID (e.g., “squad”, “openai/gsm8k”)

pub fn repo_id(&self) -> &str

Returns the repository ID.

pub fn revision(&self) -> &str

Returns the revision being used.

pub fn subset(&self) -> Option<&str>

Returns the subset/config if set.

pub fn split(&self) -> Option<&str>

Returns the split if set.

pub fn cache_dir(&self) -> &Path

Returns the cache directory.

pub fn download(&self) -> Result<ArrowDataset>

Downloads the dataset and returns an ArrowDataset.

This method:

1. Checks the local cache for existing data
2. Downloads parquet files from HuggingFace Hub if not cached
3. Loads the parquet files into an ArrowDataset

Errors

Returns an error if:

• The dataset cannot be found on HuggingFace Hub
• The download fails
• The parquet files cannot be parsed

pub fn download_to(&self, output: impl AsRef<Path>) -> Result<ArrowDataset>

Downloads the dataset to a specific output path.

Arguments

• output - Path where the dataset should be saved

Errors

Returns an error if the download or save fails.

pub fn clear_cache(&self) -> Result<()>

Clears the local cache for this dataset.

Errors

Returns an error if the cache cannot be deleted.

Trait Implementations

impl Clone for HfDataset

fn clone(&self) -> HfDataset

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HfDataset

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.