Struct TokenizerOptions 

pub struct TokenizerOptions<'a> {
    pub cls_token: Option<&'a str>,
    pub sep_token: Option<&'a str>,
}

Configuration for a Tokenizer.

Fields

cls_token: Option<&'a str>

Token added at the start of the output. For BERT models, this is the [CLS] token.

sep_token: Option<&'a str>

Token added after each encoded sequence in the output. For BERT models, this is the [SEP] token.

Trait Implementations

impl<'a> Clone for TokenizerOptions<'a>

fn clone(&self) -> TokenizerOptions<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Default for TokenizerOptions<'a>

fn default() -> TokenizerOptions<'a>

Returns the “default value” for a type.