Crate mako

source · []
Expand description

Mako is a high performance data handling library


Batching module contains several utilities for dealing with batches, such as shuffling and sorting batches

Dataloader module contains the main dataloader struct, as well as dataloader utilities

Pipeline module contains the dataflow pipeline struct, as well as all pipeline utilities

Tokenization module handles all tokenization and untokenization

Vocab module contains the mako vocab object and the functions to load different vocabularies