Function tokenize_as_ids 

pub fn tokenize_as_ids(
    text: &str,
    dictionary: &TokenDictionary,
) -> Vec<QueryToken>

Tokenizes text and returns QueryTokens directly, avoiding string allocation.

This is the primary tokenization function for query processing. Tokens are classified against the dictionary immediately:

• Known tokens → QueryToken::Known(KnownToken)
• Unknown tokens → QueryToken::Unknown
• Stopwords → QueryToken::Stopword

Returns

A vector of QueryTokens (no string allocation).