Module fts 

FTS5 query helpers: escaping, stopword stripping, and safe MATCH construction.

User input must never be passed raw to SQLite FTS5: operators like AND, OR, NEAR, and bare " can change query structure or cause errors. Natural-language prompts also contain stopwords (why, not, the) that would force AND-matches to fail when those tokens are absent from notes.

Structs

PreparedQuery

Prepared FTS query plus the significant tokens used for ranking boosts.

Functions

escape_fts5_query

Escape a user query for SQLite FTS5 MATCH (legacy helper).

is_generic_topic

True when every token is a generic overview word (or the list is empty).

prepare_search_query

Build a safe FTS5 MATCH string and ranking tokens from raw user input.

significant_tokens

Strip stopwords; if nothing remains, fall back to tokens longer than 1 char.

tokenize_query

Tokenize raw user text: split on whitespace/punctuation, lowercase, keep alnum/_/-/:/.