pub fn tokenize_fts_expression(expr: &str, cjk_mode: CjkTokenizerMode) -> StringExpand description
Tokenize a query string, preserving FTS5 operators (AND, OR, NOT, NEAR).
Non-operator parts are tokenized via tokenize_for_fts. FTS5 special
characters are stripped from each token.
Equivalent to Python’s _tokenize_fts_expression() in fts.py.