Function count_passage_tokens 

pub fn count_passage_tokens(
    tokenizer: &Tokenizer,
    text: &str,
) -> Result<usize, AppError>

Counts the tokens produced by encoding text with the passage prefix.

Prepends PASSAGE_PREFIX before tokenizing so the count reflects the actual number of tokens consumed by the embedding model.

Errors

Returns Err when the tokenizer fails to encode the input.