Skip to main content

define_datasets

Macro define_datasets 

Source
macro_rules! define_datasets {
    (
        $(
            $variant:ident {
                name: $name:literal,
                description: $description:literal,
                url: $url:literal,
                entity_types: [$($etype:literal),* $(,)?],
                language: $language:literal,
                domain: $domain:literal,
                $(license: $license:literal,)?
                $(citation: $citation:literal,)?
                $(paper_url: $paper_url:literal,)?
                $(year: $year:literal,)?
                $(format: $format:literal,)?
                $(annotation_scheme: $ann_scheme:literal,)?
                $(size_hint: $size_hint:literal,)?
                $(example: $example:literal,)?
                $(notes: $notes:literal,)?
                $(splits: [$($split:literal),* $(,)?],)?
                $(tasks: [$($task:literal),* $(,)?],)?
                $(expected_docs: $expected_docs:literal,)?
                $(sha256: $sha256:literal,)?
                $(hf_id: $hf_id:literal,)?
                $(hf_config: $hf_config:literal,)?
                $(access_status: $access_status:ident,)?
                $(mirror_url: $mirror_url:literal,)?
                $(depends_on: $depends_on:literal,)?
                categories: [$($cat:ident),* $(,)?] $(,)?
            }
        ),* $(,)?
    ) => { ... };
}
Expand description

Macro to define all datasets in a single place.

Each dataset entry has:

  • variant: The enum variant name (e.g., WikiGold)
  • name: Human-readable name (e.g., “WikiGold”)
  • description: Brief description with historical context
  • url: Download URL (empty string if requires license)
  • entity_types: Slice of entity type strings
  • language: Primary language code (ISO 639-1/3). Use "mul" for multilingual datasets.
  • domain: Domain category
  • categories: List of category flags (coref, biomedical, etc.)

Optional fields (add after domain, before categories):

  • license: SPDX identifier or common name
  • citation: “Author et al. (YYYY)” format
  • paper_url: DOI or ACL Anthology URL
  • year: Publication year (u16)
  • format: Data format string
  • annotation_scheme: BIO, BIOES, IOB2, standoff
  • size_hint: Approximate size description
  • example: Short example snippet showing annotation style
  • notes: Historical context, known issues, cultural significance
  • splits: Available splits (“train”, “dev”, “test”, “all”)
  • tasks: Supported NLP tasks (“ner”, “coref”, “re”, “el”, etc.)
  • expected_docs: Expected doc count for validation (u32)
  • sha256: Content hash for integrity verification
  • access_status: DatasetAccessibility value (public, huggingface, contact_authors, etc.)
  • mirror_url: Alternative download URL if primary is unavailable
  • depends_on: Dataset ID this depends on (e.g., GCDC depends on Yahoo L6)