Crate dataforge 

Fast dataset discovery and archive helpers for thesa and other Rust tools.

Made by Trevor Knott for Knott Dynamics.

DataForge normalizes dataset targets, queries public dataset catalogs, and archives dataset files with bounded concurrent downloads. The first-class provider is Hugging Face Datasets; Zenodo is included for DOI/record-backed research datasets.

assert_eq!(
    dataforge::parse_dataset_target_for_provider(
        "https://huggingface.co/datasets/squad",
        dataforge::DatasetProvider::Hf,
    )
    .unwrap(),
    dataforge::DatasetTarget::Dataset("squad".to_string())
);
assert_eq!(
    dataforge::DatasetProvider::from_slug("hugging-face"),
    Some(dataforge::DatasetProvider::Hf)
);

Structs

ArchiveFailure

Failed dataset or file archive entry.

ArchiveOptions

Options controlling archive writes.

ArchiveSummary

Summary returned after an archive run.

ArchivedFile

One archived or skipped file.

DataForge

Blocking dataset discovery and archive client.

DatasetDetails

Full dataset metadata plus files to archive.

DatasetFile

One downloadable file in a dataset archive.

DatasetSummary

Lightweight dataset metadata returned by discovery calls.

Enums

DataForgeError

Errors returned by DataForge.

DatasetProvider

Supported dataset providers.

DatasetTarget

Normalized dataset target requested by a user or CLI.

Functions

parse_dataset_target

Parse a Hugging Face-style dataset target.

parse_dataset_target_for_provider

Parse a dataset target using provider-specific rules.

Type Aliases

Result

Result type used by DataForge APIs.