baseforge 0.1.7

# BaseForge

BaseForge is a Rust library for fast dataset discovery and local dataset archives.

Made by Trevor Knott for Knott Dynamics. BaseForge is part of the thesa archive stack alongside `siteforge`, `modelforge`, `repoforge`, Scrin, and Aisling.

BaseForge starts with Hugging Face Datasets and also supports Zenodo records. It turns user-facing dataset targets into stable Rust enums, lists matching datasets through provider APIs, inspects downloadable files, and archives those files with bounded concurrent downloads.

## What It Does Well

- Parses dataset targets for Hugging Face and Zenodo.
- Lists namespace, search, top, and latest dataset collections.
- Inspects concrete datasets to find downloadable files.
- Archives dataset files into local directories with bounded concurrency.
- Writes a `baseforge-dataset.json` manifest beside downloaded files.
- Supports skip-existing behavior for repeatable archive runs.
- Keeps provider-specific API details behind small Rust types.

## Install

```toml
[dependencies]
baseforge = "0.1.7"
```

## Quick Start

```rust,no_run
use baseforge::{ArchiveOptions, BaseForge, DatasetProvider, parse_dataset_target_for_provider};

let target = parse_dataset_target_for_provider("top", DatasetProvider::Hf)?;
let forge = BaseForge::new(DatasetProvider::Hf)?.with_page_size(25);
let summary = forge.archive_target(
    &target,
    &ArchiveOptions {
        output: "archives/datasets".into(),
        concurrency: 8,
        skip_existing: true,
        filter: Some("squad".to_string()),
    },
)?;

println!("archived {} datasets", summary.archived);
# Ok::<(), baseforge::BaseForgeError>(())
```

## Providers

Supported providers:

- `DatasetProvider::Hf`: Hugging Face Datasets.
- `DatasetProvider::Zenodo`: Zenodo records and files.

Provider aliases accept common separators, so `hugging-face`, `hugging face`, and `hugging_face` resolve to Hugging Face.

## Accepted Targets

Hugging Face:

- `owner` as a namespace listing.
- `owner/dataset` as a concrete dataset.
- `dataset:<id>` to force a concrete dataset id such as `dataset:squad`.
- `search:<query>` to search datasets.
- `https://huggingface.co/datasets/owner/dataset`.
- `top`, `trending`, `popular`.
- `latest`, `newest`, `new`.

Zenodo:

- numeric record ids such as `12345`.
- `https://zenodo.org/records/12345`.
- any other text as a search query.
- `top`, `trending`, `popular`.
- `latest`, `newest`, `new`.

## API Surface

- `parse_dataset_target(input)`: parse a Hugging Face-style target.
- `parse_dataset_target_for_provider(input, provider)`: parse with provider-specific rules.
- `DatasetProvider::from_slug(value)`: parse provider aliases.
- `BaseForge::discover(target)`: list matching datasets.
- `BaseForge::inspect(dataset_id)`: fetch files for one dataset.
- `BaseForge::archive_target(target, options)`: discover and archive in one call.
- `BaseForge::archive_datasets(datasets, options)`: archive preselected datasets.
- `ArchiveOptions`: output, concurrency, skip-existing, and filter controls.
- `ArchiveSummary`: discovered, selected, archived, skipped, file, byte, and failure counts.
- `AGENT_GUIDE`: the embedded contents of the packaged agent usage and safety guide.

## Agent Guide

See [`AGENT_GUIDE.md`](AGENT_GUIDE.md) for exact provider target rules, token
handling, discovery pagination, dataset filtering, file concurrency, archive
and progress semantics, destructive replacement behavior, safety limits, error
handling, and examples based on the public API. The same guide is available to
consumers as `baseforge::AGENT_GUIDE` without filesystem access.

## Output Layout

BaseForge writes one directory per dataset id under the configured output root. Each dataset directory contains:

- downloaded provider files, preserving safe relative paths;
- `baseforge-dataset.json` with provider metadata and file URLs.

Example:

```text
archives/datasets/
  openai_gsm8k/
    baseforge-dataset.json
    main/train.jsonl
    main/test.jsonl
```

## Role In Thesa

`thesa` uses BaseForge as its dataset archive engine. The CLI and TUI collect a dataset target, BaseForge normalizes it, discovers matching datasets, and downloads files with concurrency while thesa handles archive roots, dry-runs, filters, skip-existing behavior, and `.thesa` sidecar manifests.

This keeps dataset scraping fast and reusable while preserving the same forge split as the rest of thesa:

- `repoforge`: repository archive discovery and refresh.
- `modelforge`: model target parsing.
- `siteforge`: website capture.
- `baseforge`: dataset discovery and archive downloads.

## Package Contents

Cargo release source files are whitelisted to avoid shipping sessions, notes, build output, or local artifacts. The source whitelist contains:

- `Cargo.toml`
- `Cargo.lock`
- `README.md`
- `AGENT_GUIDE.md`
- `LICENSE`
- `CHANGELOG.md`
- `src/**`

Cargo may add its generated manifest and version-control metadata when building the crate archive.