Module tokenizers::pre_tokenizers::byte_level

Structs

ByteLevel

Provides all the necessary steps to handle the BPE tokenization at the byte-level. Takes care of all the required processing steps to transform a UTF-8 string as needed before and after the BPE model does its job.