Struct Tokenizer

pub struct Tokenizer { /* private fields */ }

Tokenizer which converts strings into token sequences consumable by the GPT-2 model, and vice-versa.

This tokenizer loads its configuration from the original OpenAI GPT-2 encoder and vocabulary “byte-pair encoding” (BPE).

Implementations

impl Tokenizer

pub fn new(bpe_path: &str, encoder_path: &str) -> Self

Creates a new in-memory tokenizer from the BPE file at bpe_path and the character encoding file at encoder_path.

pub fn encode_to_length( &self, text: &str, token_sequence_length: usize, ) -> (Vec<i32>, usize)

Encodes text into a token sequence, truncating and/or “right-padding” the encoded token sequence to fit token_sequence_length, using PAD_TOKEN as the padding token.

The returned tuple contains (token_sequence, padding_length), where padding_length is the number of padding tokens in token_sequence. If the length of token_sequence before truncation exceeds token_sequence_length, padding_length will always be zero.

Left vs. Right Padding

Common wisdom in the ML community is to “pad-left” on natural language models like GPT-2; that is, by adding padding tokens to the front of the input tokens until they fit a required input length.

However, this method “pads-right” by adding padding tokens to the end of the input tokens.

Right-padding works because GPT-2 never looks “ahead” (to the right) of its inputs, and so the right-padding will not influence the inference results of any tokens to the left.

Conversely, left-padding on GPT-2 only works if an attention mask is used, which tells the GPT-2 model to ignore certain tokens (like the padding tokens). However, attention masking is slightly more complicated to implement (albeit more efficient); therefore, this implementation does not use it.

pub fn encode(&self, text: &str) -> Vec<i32>

Encodes text into a token sequence for consumption by the GPT-2 model.

pub fn decode(&self, token_sequence: Vec<i32>) -> String

Decodes token_sequence into text.