Skip to main content

count_passage_tokens

Function count_passage_tokens 

Source
pub fn count_passage_tokens(
    tokenizer: &Tokenizer,
    text: &str,
) -> Result<usize, AppError>