Function tokenize_query 

pub fn tokenize_query(
    text: &str,
    tokenizer: &Tokenizer,
    model_max_tokens: usize,
) -> Result<Encoding>

Tokenize a query string for embedding, truncating to model_max_tokens.

Returns an crate::backend::Encoding with input_ids, attention_mask, and token_type_ids cast to i64, ready for ONNX inference.

Errors

Returns an error if the tokenizer fails to encode the text.