Skip to main content

Module tokenizer

Module tokenizer 

Source
Expand description

Tokenizer module for TreeSearch FTS5 and search indexing.

Supports CJK (Chinese, Japanese, Korean) and English tokenization. CJK tokenization mode is configurable via CjkTokenizerMode in config.

Ported from Python treesearch/tokenizer.py and treesearch/fts.py.

Modules§

cjk
CJK-specific tokenization strategies.

Statics§

RE_HAS_CJK
Regex to detect CJK characters (Chinese, Japanese, Korean).

Functions§

has_cjk
Check whether a string contains any CJK character.
tokenize
General tokenization with optional stopword removal.
tokenize_for_fts
Tokenize text for FTS5 indexing. Returns space-separated tokens.
tokenize_fts_expression
Tokenize a query string, preserving FTS5 operators (AND, OR, NOT, NEAR).