Expand description
Tokenizer module for TreeSearch FTS5 and search indexing.
Supports CJK (Chinese, Japanese, Korean) and English tokenization.
CJK tokenization mode is configurable via CjkTokenizerMode in config.
Ported from Python treesearch/tokenizer.py and treesearch/fts.py.
Modules§
- cjk
- CJK-specific tokenization strategies.
Statics§
- RE_
HAS_ CJK - Regex to detect CJK characters (Chinese, Japanese, Korean).
Functions§
- has_cjk
- Check whether a string contains any CJK character.
- tokenize
- General tokenization with optional stopword removal.
- tokenize_
for_ fts - Tokenize text for FTS5 indexing. Returns space-separated tokens.
- tokenize_
fts_ expression - Tokenize a query string, preserving FTS5 operators (AND, OR, NOT, NEAR).