Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

A tokenizer that automatically determines the appropriate tokenizer.

Implementations

impl Tokenizer

pub fn new<T: AsRef<Path>>(path: T) -> Result<Self>

Create a tokenizer instance by specifying the path to a directory containing model files.

Trait Implementations

impl Tokenizer for Tokenizer

fn encode(&self, input: &str) -> Result<Vec<String>>

Encodes a given string into a sequence of tokens.

This function takes a reference to a string and returns a vector of token strings resulting from the tokenization process.

Arguments

• input - A reference to the string to be tokenized.

Returns

A Result containing either the vector of tokens if successful or an error if the tokenization fails.

fn decode(&self, tokens: Vec<String>) -> Result<String>

Decodes a given sequence of tokens back into a single string.

This function takes a vector of token strings and reconstructs the original string.

Arguments

• tokens - A vector of strings representing the tokens to be decoded.

Returns

A Result containing either the reconstructed string if successful or an error if the decoding fails.