Skip to main content

tokenize_fts_expression

Function tokenize_fts_expression 

Source
pub fn tokenize_fts_expression(expr: &str, cjk_mode: CjkTokenizerMode) -> String
Expand description

Tokenize a query string, preserving FTS5 operators (AND, OR, NOT, NEAR).

Non-operator parts are tokenized via tokenize_for_fts. FTS5 special characters are stripped from each token.

Equivalent to Python’s _tokenize_fts_expression() in fts.py.