Struct TokenizerBuilder 

pub struct TokenizerBuilder { /* private fields */ }

Builder for Tokenizer.

Example

use kham_core::Tokenizer;

let tok = Tokenizer::builder()
    .keep_whitespace(true)
    .build();

Implementations

impl TokenizerBuilder

pub fn dict_words(self, words: &str) -> Self

Load an additional word list from a string (newline-separated words).

Words are merged with the built-in dictionary.

pub fn keep_whitespace(self, keep: bool) -> Self

Configure whether whitespace tokens are included in the output.

Default: false (whitespace is discarded).

pub fn build(self) -> Tokenizer

Consume the builder and return a configured Tokenizer.

pub fn dict_file(self, path: &str) -> Result<Self, KhamError>

Try to load a custom word list from a file path.

Only available when the std feature is enabled.

Errors

Returns KhamError::DictLoadError if the file cannot be read.

Example

use kham_core::Tokenizer;

let tok = Tokenizer::builder()
    .dict_file("my_words.txt")
    .expect("failed to load dict")
    .build();

Trait Implementations

impl Debug for TokenizerBuilder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerBuilder

fn default() -> TokenizerBuilder

Returns the “default value” for a type.