Struct CoreTokenizerBuilder 

pub struct CoreTokenizerBuilder { /* private fields */ }

Builder that orchestrates tokenizer configuration on behalf of the bindings.

Wraps lindera_analysis::tokenizer::TokenizerBuilder; filter arguments are passed as serde_json::Value so the FFI-specific value conversion stays in each binding.

Implementations

impl CoreTokenizerBuilder

pub fn new() -> CoreResult<Self>

Creates a builder with the default (empty) configuration.

pub fn from_file(file_path: &Path) -> CoreResult<Self>

Creates a builder from a YAML configuration file.

pub fn set_mode(&mut self, mode: &str) -> CoreResult<&mut Self>

Sets the segmenter mode from a string ("normal" or "decompose").

pub fn set_dictionary(&mut self, uri: &str) -> &mut Self

Sets the segmenter dictionary URI / path.

pub fn set_user_dictionary(&mut self, uri: &str) -> &mut Self

Sets the segmenter user-dictionary URI / path.

pub fn set_keep_whitespace(&mut self, keep_whitespace: bool) -> &mut Self

Sets whether whitespace tokens are kept in the output.

pub fn set_space_penalty(&mut self, value: &Value) -> CoreResult<&mut Self>

Sets the left-space penalty (Korean; see Segmenter::space_penalty) from a value with the same meaning as segmenter.space_penalty in a YAML config:

• null restores the default: the rules the dictionary ships, if any (ko-dic ships mecab-ko-dic’s rules, so they apply unless turned off);
• false turns the penalty off;
• true requires the dictionary’s rules (building fails if it ships none);
• an object such as {"rules": [{"pos": ["JKS"], "cost": 6000}]} applies those rules instead.

Arguments

• value - The setting, converted from the binding’s native value.

Returns

A mutable reference to self, for chaining, or an invalid-argument error when value is not one of the forms above.

pub fn append_character_filter(&mut self, kind: &str, args: &Value) -> &mut Self

Appends a character filter identified by kind with JSON args.

pub fn append_token_filter(&mut self, kind: &str, args: &Value) -> &mut Self

Appends a token filter identified by kind with JSON args.

pub fn build(&self) -> CoreResult<CoreTokenizer>

Builds a CoreTokenizer from the current configuration.