baseforge 0.1.0

# BaseForge

BaseForge is a Rust library for fast dataset discovery and local dataset archives.

Made by Trevor Knott for Knott Dynamics. BaseForge is part of the thesa archive stack alongside `siteforge`, `modelforge`, `repoforge`, Scrin, and Aisling.

BaseForge starts with Hugging Face Datasets and also supports Zenodo records. It turns user-facing dataset targets into stable Rust enums, lists matching datasets through provider APIs, inspects downloadable files, and archives those files with bounded concurrent downloads.

## What It Does Well

- Parses dataset targets for Hugging Face and Zenodo.
- Lists namespace, search, top, and latest dataset collections.
- Inspects concrete datasets to find downloadable files.
- Archives dataset files into local directories with bounded concurrency.
- Writes a `baseforge-dataset.json` manifest beside downloaded files.
- Supports skip-existing behavior for repeatable archive runs.
- Keeps provider-specific API details behind small Rust types.

## Install

```toml
[dependencies]
baseforge = "0.1"
```

## Quick Start

```rust,no_run
use baseforge::{ArchiveOptions, BaseForge, DatasetProvider, parse_dataset_target_for_provider};

let target = parse_dataset_target_for_provider("top", DatasetProvider::Hf)?;
let forge = BaseForge::new(DatasetProvider::Hf)?.with_page_size(25);
let summary = forge.archive_target(
    &target,
    &ArchiveOptions {
        output: "archives/datasets".into(),
        concurrency: 8,
        skip_existing: true,
        filter: Some("squad".to_string()),
    },
)?;

println!("archived {} datasets", summary.archived);
# Ok::<(), baseforge::BaseForgeError>(())
```

## Providers

Supported providers:

- `DatasetProvider::Hf`: Hugging Face Datasets.
- `DatasetProvider::Zenodo`: Zenodo records and files.

Provider aliases accept common separators, so `hugging-face`, `hugging face`, and `hugging_face` resolve to Hugging Face.

## Accepted Targets

Hugging Face:

- `owner` as a namespace listing.
- `owner/dataset` as a concrete dataset.
- `dataset:<id>` to force a concrete dataset id such as `dataset:squad`.
- `search:<query>` to search datasets.
- `https://huggingface.co/datasets/owner/dataset`.
- `top`, `trending`, `popular`.
- `latest`, `newest`, `new`.

Zenodo:

- numeric record ids such as `12345`.
- `https://zenodo.org/records/12345`.
- any other text as a search query.
- `top`, `trending`, `popular`.
- `latest`, `newest`, `new`.

## API Surface

- `parse_dataset_target(input)`: parse a Hugging Face-style target.
- `parse_dataset_target_for_provider(input, provider)`: parse with provider-specific rules.
- `DatasetProvider::from_slug(value)`: parse provider aliases.
- `BaseForge::discover(target)`: list matching datasets.
- `BaseForge::inspect(dataset_id)`: fetch files for one dataset.
- `BaseForge::archive_target(target, options)`: discover and archive in one call.
- `BaseForge::archive_datasets(datasets, options)`: archive preselected datasets.
- `ArchiveOptions`: output, concurrency, skip-existing, and filter controls.
- `ArchiveSummary`: discovered, selected, archived, skipped, file, byte, and failure counts.

## Output Layout

BaseForge writes one directory per dataset id under the configured output root. Each dataset directory contains:

- downloaded provider files, preserving safe relative paths;
- `baseforge-dataset.json` with provider metadata and file URLs.

Example:

```text
archives/datasets/
  openai_gsm8k/
    baseforge-dataset.json
    main/train.jsonl
    main/test.jsonl
```

## Role In Thesa

`thesa` uses BaseForge as its dataset archive engine. The CLI and TUI collect a dataset target, BaseForge normalizes it, discovers matching datasets, and downloads files with concurrency while thesa handles archive roots, dry-runs, filters, skip-existing behavior, and `.thesa` sidecar manifests.

This keeps dataset scraping fast and reusable while preserving the same forge split as the rest of thesa:

- `repoforge`: repository archive discovery and refresh.
- `modelforge`: model target parsing.
- `siteforge`: website capture.
- `baseforge`: dataset discovery and archive downloads.

## Package Contents

Cargo releases are whitelisted to avoid shipping sessions, notes, build output, or local artifacts. The package contains only:

- `Cargo.toml`
- `Cargo.lock`
- `README.md`
- `LICENSE`
- `CHANGELOG.md`
- `src/**`