pub struct Fts5Tokenizer {
pub tokenizer: Tokenizer,
}Expand description
Wrapper for Lindera tokenizer used in FTS5.
This structure wraps the Lindera Tokenizer for use in the FTS5 tokenizer API.
Each FTS5 table using the Lindera tokenizer will have its own instance of this struct.
§Memory Management
Instances are heap-allocated in fts5_create_lindera_tokenizer
and deallocated in fts5_delete_lindera_tokenizer.
Fields§
§tokenizer: TokenizerThe underlying Lindera tokenizer instance.