Struct Fts5Tokenizer 

pub struct Fts5Tokenizer {
    pub tokenizer: Tokenizer,
}

Wrapper for Lindera tokenizer used in FTS5.

This structure wraps the Lindera Tokenizer for use in the FTS5 tokenizer API. Each FTS5 table using the Lindera tokenizer will have its own instance of this struct.

Memory Management

Instances are heap-allocated in fts5_create_lindera_tokenizer and deallocated in fts5_delete_lindera_tokenizer.

Fields

tokenizer: Tokenizer

The underlying Lindera tokenizer instance.