pub fn count_passage_tokens(text: &str) -> Result<usize, AppError>
Returns the approximate token count for text when prefixed with prefix (e.g. passage: for embed_passage).
text
prefix
passage:
embed_passage