Struct DatasetLoader 

pub struct DatasetLoader { /* private fields */ }

Loads and caches NER datasets.

Caching Strategy (Tiered)

1. Local cache (~/.cache/anno/datasets/): Checked first, fastest
2. S3 cache (s3://anno-data/): Checked if ANNO_S3_CACHE=1 and AWS credentials available
3. URL download: Original source URL, last resort

This tiered approach:

• Maximizes speed (local cache hit is instant)
• Provides redundancy (S3 backup if original URLs go down)
• Allows sharing datasets across machines via S3

Environment Variables

• ANNO_CACHE_DIR: Override default cache location
• ANNO_S3_CACHE: Set to “1” to enable S3 fallback (requires AWS credentials)
• ANNO_S3_BUCKET: Override S3 bucket name (default: arc-anno-data)

Example

# Enable S3 caching
export ANNO_S3_CACHE=1
export AWS_PROFILE=default  # or set AWS_ACCESS_KEY_ID, etc.

# Now load_or_download() will try S3 before URL

Implementations

impl DatasetLoader

pub fn new() -> Result<Self>

Create a new loader with default cache directory.

Default location: ~/.cache/anno/datasets (platform cache via dirs crate) Falls back to .anno/datasets in current directory if dirs crate unavailable.

S3 fallback is enabled if ANNO_S3_CACHE=1 environment variable is set.

pub fn with_cache_dir(cache_dir: impl Into<PathBuf>) -> Result<Self>

Create a loader with a custom cache directory.

pub fn cache_dir(&self) -> &Path

Base directory for cached datasets.

pub fn s3_enabled(&self) -> bool

Whether S3 fallback caching is enabled.

pub fn s3_bucket(&self) -> Option<&str>

The configured S3 bucket name (if enabled).

pub fn cached_manifest_entries(&self) -> Vec<CacheManifestEntry>

Snapshot of cached dataset manifest entries (cloned).

This is intended for tooling (CLI/scripts) that wants to iterate cached datasets without holding the manifest lock for a long time.

pub fn upload_cached_dataset_to_s3( &self, bucket: &str, id: DatasetId, ) -> Result<()>

Upload a locally cached dataset (by DatasetId) to S3, using its manifest entry.

This uses the same object layout as load_or_download()’s best-effort upload:

• datasets/<cache_filename> (legacy/mutable key)
• datasets/by-sha256/<sha>/<cache_filename> (immutable snapshot)
• datasets/<cache_filename>.latest.json (pointer)
• datasets/<cache_filename>.manifest.json (sidecar metadata)

pub fn cache_path(&self, id: LoadableDatasetId) -> PathBuf

Get the cache path for a dataset.

pub fn is_cached(&self, id: LoadableDatasetId) -> bool

Check if a dataset is cached locally.

pub fn load(&self, id: LoadableDatasetId) -> Result<LoadedDataset>

Load a dataset from cache.

Returns an error if the dataset is not cached.

pub fn load_or_download(&self, id: LoadableDatasetId) -> Result<LoadedDataset>

Load or download a dataset.

Tries cache first, then S3 (if enabled), then downloads from URL.

pub fn parse_content_str( &self, content: &str, id: DatasetId, ) -> Result<LoadedDataset>

Parse content based on dataset format.

This is intentionally pub (behind the eval module) to enable offline evaluation workflows and integration tests that want to exercise parsing without network I/O.

It does not download or read from cache. For typical usage, prefer DatasetLoader::load_or_download / DatasetLoader::load.

pub fn load_coref(&self, id: DatasetId) -> Result<Vec<CorefDocument>>

Load coreference dataset, returning documents with chains.

Use this for GAP, PreCo, and LitBank datasets.

pub fn load_or_download_coref( &self, id: DatasetId, ) -> Result<Vec<CorefDocument>>

Load coreference dataset, downloading if needed.

pub fn load_relation(&self, id: DatasetId) -> Result<Vec<RelationDocument>>

Load relation extraction dataset, returning documents with relations.

Use this for DocRED and ReTACRED datasets.

pub fn load_or_download_relation( &self, id: DatasetId, ) -> Result<Vec<RelationDocument>>

Load relation extraction dataset, downloading if needed.

pub fn load_all_cached(&self) -> Vec<(DatasetId, Result<LoadedDataset>)>

Load all cached datasets.

pub fn status(&self) -> Vec<(DatasetId, bool)>

Get status of all datasets.

Trait Implementations

impl Default for DatasetLoader

fn default() -> Self

Returns the “default value” for a type.