Trait llm_chain::tokens::Tokenizer

source · 
pub trait Tokenizer<TokenType: Clone> {
    // Required methods
    fn tokenize_str(&self, doc: &str) -> Result<Vec<TokenType>, TokenizerError>;
    fn to_string(
        &self,
        tokens: Vec<TokenType>
    ) -> Result<String, TokenizerError>;
}

Required Methods§

source

fn tokenize_str(&self, doc: &str) -> Result<Vec<TokenType>, TokenizerError>

Tokenizes a string.

Parameters
  • doc: The string to tokenize.
Returns

A Result containing a vector of tokens, or an error if there was a problem.

source

fn to_string(&self, tokens: Vec<TokenType>) -> Result<String, TokenizerError>

Converts a vector of tokens into a string.

Parameters
  • tokens: The slice of tokens to convert.
Returns

A Result containing a string, or an error if there was a problem.

Implementors§