Module oscar_io::oscar_doc

source ·
Expand description

OSCAR Schema v2 (OSCAR 22.01) types, readers and writers.

Each document is materialized by a Document, holding Metadata, WarcHeaders and content (that is a String).

Structs

  • Document
    A Document is a structure holding content, WARC headers and OSCAR-specific metadata.
  • Metadata
    OSCAR Metadata. Contains document identification, annotations and sentence-level identifications.
  • Reader
    Document reader. The inner type has to implement BufRead.
  • SplitFolderReader
  • SplitReader
    In the case where we have multiple splits for a given subcorpus.
  • Writer

Type Definitions

  • WarcHeaders
    Simple alias to a HashMap<String, String> for simplifying the sourcecode.