Crate baseforge 

Fast dataset discovery and archive helpers for thesa and other Rust tools.

Made by Trevor Knott for Knott Dynamics.

BaseForge normalizes dataset targets, queries public dataset catalogs, and archives dataset files with bounded concurrent downloads. The first-class provider is Hugging Face Datasets; Zenodo is included for DOI/record-backed research datasets.

assert_eq!(
    baseforge::parse_dataset_target_for_provider(
        "https://huggingface.co/datasets/squad",
        baseforge::DatasetProvider::Hf,
    )
    .unwrap(),
    baseforge::DatasetTarget::Dataset("squad".to_string())
);
assert_eq!(
    baseforge::DatasetProvider::from_slug("hugging-face"),
    Some(baseforge::DatasetProvider::Hf)
);

Structs

ArchiveFailure

Failed dataset or file archive entry.

ArchiveOptions

Options controlling archive writes.

ArchiveProgress

Snapshot emitted during dataset archive downloads.

ArchiveProgressFile

File currently being archived by a worker.

ArchiveSummary

Summary returned after an archive run.

ArchivedFile

One archived or skipped file.

BaseForge

Blocking dataset discovery and archive client.

DatasetDetails

Full dataset metadata plus files to archive.

DatasetFile

One downloadable file in a dataset archive.

DatasetSummary

Lightweight dataset metadata returned by discovery calls.

Enums

ArchiveProgressPhase

Phase reported by dataset archive progress snapshots.

BaseForgeError

Errors returned by BaseForge.

DatasetProvider

Supported dataset providers.

DatasetTarget

Normalized dataset target requested by a user or CLI.

Functions

parse_dataset_target

Parse a Hugging Face-style dataset target.

parse_dataset_target_for_provider

Parse a dataset target using provider-specific rules.

Type Aliases

Result

Result type used by BaseForge APIs.