Struct CoreTokenizerBuilder 

pub struct CoreTokenizerBuilder { /* private fields */ }

Builder that orchestrates tokenizer configuration on behalf of the bindings.

Wraps lindera::tokenizer::TokenizerBuilder; filter arguments are passed as serde_json::Value so the FFI-specific value conversion stays in each binding.

Implementations

impl CoreTokenizerBuilder

pub fn new() -> CoreResult<Self>

Creates a builder with the default (empty) configuration.

pub fn from_file(file_path: &Path) -> CoreResult<Self>

Creates a builder from a YAML configuration file.

pub fn set_mode(&mut self, mode: &str) -> CoreResult<&mut Self>

Sets the segmenter mode from a string ("normal" or "decompose").

pub fn set_dictionary(&mut self, uri: &str) -> &mut Self

Sets the segmenter dictionary URI / path.

pub fn set_user_dictionary(&mut self, uri: &str) -> &mut Self

Sets the segmenter user-dictionary URI / path.

pub fn set_keep_whitespace(&mut self, keep_whitespace: bool) -> &mut Self

Sets whether whitespace tokens are kept in the output.

pub fn append_character_filter(&mut self, kind: &str, args: &Value) -> &mut Self

Appends a character filter identified by kind with JSON args.

pub fn append_token_filter(&mut self, kind: &str, args: &Value) -> &mut Self

Appends a token filter identified by kind with JSON args.

pub fn build(&self) -> CoreResult<CoreTokenizer>

Builds a CoreTokenizer from the current configuration.