Struct BpeConfig 

pub struct BpeConfig {
    pub unk_token: String,
    pub bos_token: Option<String>,
    pub eos_token: Option<String>,
    pub pad_token: Option<String>,
    pub add_prefix_space: bool,
    pub vocab_size: usize,
}

BPE tokenizer configuration.

Fields

unk_token: String

Unknown token

bos_token: Option<String>

Beginning of sentence token

eos_token: Option<String>

End of sentence token

pad_token: Option<String>

Padding token

add_prefix_space: bool

Whether to add prefix space

vocab_size: usize

Maximum vocabulary size

Implementations

impl BpeConfig

pub fn whisper() -> Self

Create config for Whisper tokenizer

pub fn gpt2() -> Self

Create config for GPT-2 tokenizer

pub fn llama() -> Self

Create config for Llama tokenizer

pub fn qwen2() -> Self

Create config for Qwen2 tokenizer (GH-128)

Qwen2 uses a 151936 token vocabulary with special chat tokens.

Trait Implementations

impl Clone for BpeConfig

fn clone(&self) -> BpeConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BpeConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for BpeConfig

fn default() -> Self

Returns the “default value” for a type.