# BaseForge Agent Guide
This guide describes the public API and operational behavior of BaseForge
0.1.7. BaseForge is a blocking Rust client for discovering, inspecting, and
archiving Hugging Face datasets and Zenodo records.
The same text is available at runtime as `baseforge::AGENT_GUIDE`.
## Choose A Provider And Parse A Target
Create one `BaseForge` per `DatasetProvider`. Parse input with the same provider
that the client uses:
```rust,no_run
use baseforge::{BaseForge, DatasetProvider, parse_dataset_target_for_provider};
let provider = DatasetProvider::Hf;
let target = parse_dataset_target_for_provider(
"https://huggingface.co/datasets/openai/gsm8k",
provider,
)?;
let forge = BaseForge::new(provider)?;
let datasets = forge.discover(&target)?;
println!("found {} dataset(s)", datasets.len());
# Ok::<(), baseforge::BaseForgeError>(())
```
`parse_dataset_target(input)` is shorthand for Hugging Face parsing.
`DatasetProvider::from_slug` accepts `hf`, `huggingface`,
`hugging-face`, `hugging_face`, `hugging face`, `huggingfacedatasets`,
`hfdatasets`, `zen`, and `zenodo`, ignoring separators and ASCII case.
The following aliases are recognized before provider-specific parsing:
- `top`, `trending`, and `popular` produce `DatasetTarget::Top`.
- `latest`, `newest`, and `new` produce `DatasetTarget::Latest`.
- `dataset:<value>` forces `DatasetTarget::Dataset`.
- `search:<value>` forces `DatasetTarget::Search`.
- `org:<value>`, `owner:<value>`, and `namespace:<value>` force
`DatasetTarget::Namespace`.
Preset aliases are ASCII case-insensitive. The explicit prefixes are
case-sensitive and must be lowercase. Empty input and empty explicit values are
`BaseForgeError::InvalidTarget` errors.
### Hugging Face Targets
- A bare value with one path component, such as `openai`, is a namespace.
- A bare value with at least two path components, such as `openai/gsm8k`, is a
concrete dataset. Only the first two components are retained.
- Bare input containing whitespace is a search query.
- `dataset:squad` forces a concrete single-component dataset id.
- Hugging Face dataset URLs using `http` or `https`, optional `www`, and either
`huggingface.co` or `hf.co` are accepted. Query strings and fragments are
discarded. A one-component dataset URL is a concrete dataset.
Examples include `openai`, `openai/gsm8k`, `search:code data`,
`dataset:squad`, and `https://huggingface.co/datasets/openai/gsm8k`.
### Zenodo Targets
- An all-digit value is a concrete record id.
- `http` and `https` URLs under `/record/<id>` or `/records/<id>`, with optional
`www`, are concrete records. Text after the id is discarded.
- Any other unprefixed text is a search query.
- `dataset:<value>` can force a concrete record value.
- Namespace targets are accepted, but Zenodo discovery treats them exactly like
search queries.
Examples include `12345`, `https://zenodo.org/records/12345`, and
`climate data`.
## Tokens
`BaseForge::with_token` accepts a token string and ignores strings that are
empty after trimming. BaseForge does not read token files or environment
variables; the caller must obtain and pass the token. The original, untrimmed
non-empty string is retained.
For Hugging Face, the token is sent as `Authorization: Bearer <token>` on API
discovery/inspection requests and file download requests. This permits access
to repositories the token is authorized to read.
Zenodo discovery and inspection do not use the configured token. The generic
archive downloader does add a configured token to file requests for either
provider, including URLs supplied by Zenodo. In normal use, configure tokens
only for Hugging Face and only pass trusted provider data to archiving. Never
log the token or place it in a target, filter, output path, or manifest field.
```rust,no_run
use baseforge::{BaseForge, DatasetProvider};
let mut forge = BaseForge::new(DatasetProvider::Hf)?;
if let Ok(token) = std::env::var("HF_TOKEN") {
forge = forge.with_token(token);
}
# Ok::<(), baseforge::BaseForgeError>(())
```
## Discovery And Inspection
`BaseForge::discover(&target)` returns `Vec<DatasetSummary>`:
- Hugging Face `Dataset` inspects that dataset. `Namespace` queries the API's
`author` field. `Search` queries `search`. `Top` sorts all listed datasets by
downloads descending. `Latest` sorts by `lastModified` descending.
- Zenodo `Dataset` inspects that record. `Namespace` and `Search` use Zenodo's
`q` parameter and `mostviewed` sort. `Top` uses `mostviewed`; `Latest` uses
`newest`.
List discovery fetches pages until a page contains fewer than `page_size`
items. `BaseForge::with_page_size(n)` sets that request size and clamps zero to
one; the default is 100. It does not limit the total result count, deduplicate
results, or stop after one page. Broad `Top`, `Latest`, or search targets can
therefore perform many requests and retain a large result vector.
`BaseForge::inspect(dataset_id)` returns `DatasetDetails`, containing a summary
and downloadable files. Hugging Face includes every sibling whose path is not
blank and generates a `resolve/main` URL. Zenodo includes each file that has a
`self` or `download` link, preferring `self`. Provider-reported sizes and
checksums are metadata and may be absent.
## Selection And Concurrency
`ArchiveOptions::filter` is a dataset filter, not a file filter. A non-empty
filter selects summaries whose dataset id or optional title contains the
filter, using case-insensitive substring matching. Whitespace around the filter
is ignored. An empty or whitespace-only filter selects every supplied summary.
There is no public file path, extension, glob, size, or checksum filter. After
inspection, every eligible file returned by the provider is queued. Filter or
inspect `DatasetSummary`/`DatasetDetails` in caller code when stricter policy is
required; the public archive methods themselves re-inspect supplied summaries.
`ArchiveOptions::concurrency` bounds file download worker threads across all
selected datasets. The default is four. Zero is treated as one, and the worker
count never exceeds the number of queued files. Dataset inspection, directory
creation, and manifest writes happen sequentially before file workers start.
The API and worker threads are blocking, not async.
## Archive Behavior
Use `archive_target` to discover and archive in one call, or discover first and
pass summaries to `archive_datasets` or `archive_datasets_with_progress`.
```rust,no_run
use baseforge::{ArchiveOptions, BaseForge, DatasetProvider, DatasetTarget};
let forge = BaseForge::new(DatasetProvider::Hf)?;
let datasets = forge.discover(&DatasetTarget::Dataset(
"openai/gsm8k".to_string(),
))?;
let options = ArchiveOptions {
output: "archives/datasets".into(),
concurrency: 4,
skip_existing: true,
filter: None,
};
let summary = forge.archive_datasets_with_progress(
&datasets,
&options,
|progress| {
eprintln!(
"{}: {}/{} files, {} bytes",
progress.phase.as_str(),
progress.completed_files,
progress.total_files,
progress.bytes_written,
);
},
)?;
if !summary.is_success() {
for failure in &summary.failed {
eprintln!("{} {:?}: {}", failure.dataset_id, failure.file, failure.message);
}
}
# Ok::<(), baseforge::BaseForgeError>(())
```
The output root is created if needed and must be a directory. Each selected
dataset is written under a sanitized dataset id, for example `openai/gsm8k`
becomes `openai_gsm8k`.
Existing paths require special care:
- With `skip_existing: true`, any existing dataset path is treated as a
completed dataset and skipped without inspection or validation. BaseForge
does not check its manifest or files and does not add per-file skip entries.
- With `skip_existing: false` (the default), any existing dataset path is
deleted first. Directories are removed recursively and non-directories are
removed as files. This is destructive replacement, not refresh or resume.
After successful inspection, BaseForge writes `baseforge-dataset.json` before
downloads start. It is pretty-printed `DatasetDetails` JSON and includes the
summary plus source paths, direct download URLs, sizes, and provider checksums.
Checksums are recorded but not verified. A dataset with no queued files is
counted as archived after its manifest is written.
The dataset directory is created before inspection. If inspection fails, that
directory remains and has no newly written manifest. A later run with
`skip_existing: true` will still skip it solely because the path exists, so
callers that retry failures should remove or validate failed output paths first.
Downloads stream to a sibling temporary path whose extension ends in
`.baseforge-part`, validate `Content-Length` when the response supplies it,
flush, and rename the temporary file into place. Failed streaming writes remove
the partial file on a best-effort basis. There is no resume support.
`ArchiveSummary` reports input `discovered`, post-filter `selected`, completed
dataset `archived`, whole-directory `skipped`, newly written
`files_archived`, `bytes_written`, successful file records, and failures.
`attempted()` is archived plus skipped plus unique failed datasets.
`is_success()` means the failure list is empty. A dataset is archived only when
all its queued file downloads succeed.
## Progress Semantics
Only `archive_datasets_with_progress` accepts a progress callback. It emits:
- one `Starting` snapshot before workers start;
- `Downloading` snapshots for worker starts, byte advances, and finishes;
- one `Completed` snapshot after the event channel closes and workers join.
Byte advances occur at download start, approximately every MiB, and at the end
of a stream. `bytes_written` includes completed successful bytes plus current
bytes for all active downloads. `active_file`, `active_bytes_written`, and
`active_total_bytes` describe only the first active worker even when multiple
files are downloading. `completed_files` counts both successful and failed file
attempts. No snapshots are emitted when no files are queued.
The callback runs synchronously while the archive call coordinates worker
events. Keep it quick and do not panic.
## Path Safety And Collisions
Provider ids and file paths are treated as untrusted names. BaseForge retains
ASCII letters, digits, `-`, `_`, `.`, and `/`, replaces other characters with
`_`, trims surrounding whitespace and underscores after sanitizing, removes
empty, `.` and `..` path components, and falls back to `dataset.bin` for an
empty file path. This prevents provider paths from directly escaping the
dataset directory.
Sanitization is not uniqueness enforcement. Different ids or paths can map to
the same local path, absolute-looking paths are made relative, and
platform-reserved names are not specially handled. Apply caller-side policy
when archiving untrusted or adversarial catalogs, and use a dedicated output
root that contains no unrelated data because replacement can recursively delete
dataset paths.
## Limits And Operational Policy
BaseForge does not currently provide request timeouts, retries, backoff,
rate-limit handling, cancellation, total result limits, file count limits,
download byte limits, free-space checks, signature checks, checksum validation,
or transactional rollback of a complete dataset. The default HTTP client may
follow redirects. Broad discovery and large files can consume substantial
network, memory, disk, time, and provider quota.
Agents should establish their own policy before calling archive methods:
- Prefer concrete dataset/record targets or inspect broad discovery results
before archiving.
- Enforce allowlists and result, file, and byte limits in caller code.
- Confirm the dedicated output root and `skip_existing` choice explicitly.
- Use conservative concurrency and respect provider rate limits.
- Treat manifest URLs, descriptions, tags, error bodies, and provider metadata
as untrusted data.
- Check available disk space and retain failures for audit or retry decisions.
## Errors
Public methods return `baseforge::Result<T>` with `BaseForgeError`:
- `InvalidTarget` covers empty or malformed targets.
- `UnsupportedTarget` is part of the public error type but is not currently
produced by the implemented provider paths.
- `ProviderApi` includes the provider slug, HTTP status, and response body.
- `NotFound` represents a 404 from concrete Hugging Face or Zenodo inspection.
- `DownloadFailed` is part of the public error type; archive workers currently
report equivalent details through `ArchiveFailure` instead.
- `OutputNotDirectory`, `Io`, `Request`, and `Json` cover setup, filesystem,
HTTP transport, and serialization failures.
Discovery and direct inspection return provider/transport/JSON errors directly.
Archive setup, destructive replacement, directory creation, and manifest write
failures return `Err` and stop the call. An individual dataset inspection or
file download failure is instead appended to `ArchiveSummary::failed`; the
archive method can return `Ok(summary)` with failures. Always inspect
`summary.is_success()` or `summary.failed`.
Provider API and download failures may contain response bodies. Avoid exposing
those diagnostics to untrusted users without redaction and size controls.