Crate dataflow_nlp

source ·

Modules

  • batching
    Uilities for dealing with batches, such as shuffling and sorting batches
  • pipelines
    Dataflow pipeline nodes
  • tokenization
    All tokenization and untokenization
  • vocab
    Vocab object and the functions to load different vocabularies