macro_rules! define_datasets {
(
$(
$variant:ident {
name: $name:literal,
description: $description:literal,
url: $url:literal,
entity_types: [$($etype:literal),* $(,)?],
language: $language:literal,
domain: $domain:literal,
$(license: $license:literal,)?
$(citation: $citation:literal,)?
$(paper_url: $paper_url:literal,)?
$(year: $year:literal,)?
$(format: $format:literal,)?
$(annotation_scheme: $ann_scheme:literal,)?
$(size_hint: $size_hint:literal,)?
$(example: $example:literal,)?
$(notes: $notes:literal,)?
$(splits: [$($split:literal),* $(,)?],)?
$(tasks: [$($task:literal),* $(,)?],)?
$(expected_docs: $expected_docs:literal,)?
$(sha256: $sha256:literal,)?
$(hf_id: $hf_id:literal,)?
$(hf_config: $hf_config:literal,)?
$(access_status: $access_status:ident,)?
$(mirror_url: $mirror_url:literal,)?
$(depends_on: $depends_on:literal,)?
categories: [$($cat:ident),* $(,)?] $(,)?
}
),* $(,)?
) => { ... };
}Expand description
Macro to define all datasets in a single place.
Each dataset entry has:
variant: The enum variant name (e.g.,WikiGold)name: Human-readable name (e.g., “WikiGold”)description: Brief description with historical contexturl: Download URL (empty string if requires license)entity_types: Slice of entity type stringslanguage: Primary language code (ISO 639-1/3). Use"mul"for multilingual datasets.domain: Domain categorycategories: List of category flags (coref, biomedical, etc.)
Optional fields (add after domain, before categories):
license: SPDX identifier or common namecitation: “Author et al. (YYYY)” formatpaper_url: DOI or ACL Anthology URLyear: Publication year (u16)format: Data format stringannotation_scheme: BIO, BIOES, IOB2, standoffsize_hint: Approximate size descriptionexample: Short example snippet showing annotation stylenotes: Historical context, known issues, cultural significancesplits: Available splits (“train”, “dev”, “test”, “all”)tasks: Supported NLP tasks (“ner”, “coref”, “re”, “el”, etc.)expected_docs: Expected doc count for validation (u32)sha256: Content hash for integrity verificationaccess_status: DatasetAccessibility value (public, huggingface, contact_authors, etc.)mirror_url: Alternative download URL if primary is unavailabledepends_on: Dataset ID this depends on (e.g., GCDC depends on Yahoo L6)