baseforge 0.1.5

Fast dataset discovery and archive helpers for Hugging Face and Zenodo
Documentation

BaseForge

BaseForge is a Rust library for fast dataset discovery and local dataset archives.

Made by Trevor Knott for Knott Dynamics. BaseForge is part of the thesa archive stack alongside siteforge, modelforge, repoforge, Scrin, and Aisling.

BaseForge starts with Hugging Face Datasets and also supports Zenodo records. It turns user-facing dataset targets into stable Rust enums, lists matching datasets through provider APIs, inspects downloadable files, and archives those files with bounded concurrent downloads.

What It Does Well

  • Parses dataset targets for Hugging Face and Zenodo.
  • Lists namespace, search, top, and latest dataset collections.
  • Inspects concrete datasets to find downloadable files.
  • Archives dataset files into local directories with bounded concurrency.
  • Writes a baseforge-dataset.json manifest beside downloaded files.
  • Supports skip-existing behavior for repeatable archive runs.
  • Keeps provider-specific API details behind small Rust types.

Install

[dependencies]
baseforge = "0.1"

Quick Start

use baseforge::{ArchiveOptions, BaseForge, DatasetProvider, parse_dataset_target_for_provider};

let target = parse_dataset_target_for_provider("top", DatasetProvider::Hf)?;
let forge = BaseForge::new(DatasetProvider::Hf)?.with_page_size(25);
let summary = forge.archive_target(
    &target,
    &ArchiveOptions {
        output: "archives/datasets".into(),
        concurrency: 8,
        skip_existing: true,
        filter: Some("squad".to_string()),
    },
)?;

println!("archived {} datasets", summary.archived);
# Ok::<(), baseforge::BaseForgeError>(())

Providers

Supported providers:

  • DatasetProvider::Hf: Hugging Face Datasets.
  • DatasetProvider::Zenodo: Zenodo records and files.

Provider aliases accept common separators, so hugging-face, hugging face, and hugging_face resolve to Hugging Face.

Accepted Targets

Hugging Face:

  • owner as a namespace listing.
  • owner/dataset as a concrete dataset.
  • dataset:<id> to force a concrete dataset id such as dataset:squad.
  • search:<query> to search datasets.
  • https://huggingface.co/datasets/owner/dataset.
  • top, trending, popular.
  • latest, newest, new.

Zenodo:

  • numeric record ids such as 12345.
  • https://zenodo.org/records/12345.
  • any other text as a search query.
  • top, trending, popular.
  • latest, newest, new.

API Surface

  • parse_dataset_target(input): parse a Hugging Face-style target.
  • parse_dataset_target_for_provider(input, provider): parse with provider-specific rules.
  • DatasetProvider::from_slug(value): parse provider aliases.
  • BaseForge::discover(target): list matching datasets.
  • BaseForge::inspect(dataset_id): fetch files for one dataset.
  • BaseForge::archive_target(target, options): discover and archive in one call.
  • BaseForge::archive_datasets(datasets, options): archive preselected datasets.
  • ArchiveOptions: output, concurrency, skip-existing, and filter controls.
  • ArchiveSummary: discovered, selected, archived, skipped, file, byte, and failure counts.

Output Layout

BaseForge writes one directory per dataset id under the configured output root. Each dataset directory contains:

  • downloaded provider files, preserving safe relative paths;
  • baseforge-dataset.json with provider metadata and file URLs.

Example:

archives/datasets/
  openai_gsm8k/
    baseforge-dataset.json
    main/train.jsonl
    main/test.jsonl

Role In Thesa

thesa uses BaseForge as its dataset archive engine. The CLI and TUI collect a dataset target, BaseForge normalizes it, discovers matching datasets, and downloads files with concurrency while thesa handles archive roots, dry-runs, filters, skip-existing behavior, and .thesa sidecar manifests.

This keeps dataset scraping fast and reusable while preserving the same forge split as the rest of thesa:

  • repoforge: repository archive discovery and refresh.
  • modelforge: model target parsing.
  • siteforge: website capture.
  • baseforge: dataset discovery and archive downloads.

Package Contents

Cargo releases are whitelisted to avoid shipping sessions, notes, build output, or local artifacts. The package contains only:

  • Cargo.toml
  • Cargo.lock
  • README.md
  • LICENSE
  • CHANGELOG.md
  • src/**