Function load_tokenizer 

pub fn load_tokenizer() -> Result<Tokenizer, c_int>

Loads and initializes a Lindera tokenizer.

This function creates a new Lindera tokenizer using the configuration specified by the LINDERA_CONFIG_PATH environment variable. The configuration file controls segmentation mode, character filters, and token filters.

Returns

• Ok(Tokenizer) - Successfully initialized tokenizer
• Err(c_int) - Returns SQLITE_INTERNAL if tokenizer creation fails

Errors

This function will return an error if:

• The tokenizer builder cannot be created (e.g., missing or invalid configuration)
• The tokenizer cannot be built from the builder

Error messages are written to stderr for debugging purposes.

Examples

Set the configuration path environment variable before loading:

export LINDERA_CONFIG_PATH=./resources/lindera.yml

Then load the tokenizer:

let tokenizer = load_tokenizer().expect("Failed to load tokenizer");