Struct JaKoTokenizer 

pub struct JaKoTokenizer { /* private fields */ }

Splits phrases using Japanese/Korean word segmentation.

Each token emitted by this will also contain JaKoTokenMetadata in its Token::metadata.

This uses lindera::tokenizer::Tokenizer.

Implementations

impl JaKoTokenizer

pub fn from_dictionary( dictionary: Dictionary, user_dictionary: Option<UserDictionary>, ) -> Self

Creates a new Japanese/Korean tokenizer using the given lindera::dictionary::Dictionary and default segmenter settings.

pub fn from_segmenter(segmenter: Segmenter) -> Self

Creates a new Japanese/Korean tokenizer using the given lindera::segmenter::Segmenter.

Trait Implementations

impl Tokenizer for JaKoTokenizer

fn tokenize<'a>(&'a self, s: &'a str) -> impl Iterator<Item = Token<'a>>

Tokenizes the input, returning an iterator over the tokens.

fn filter<F>(self, filter: F) -> impl Tokenizer

where Self: Sized, F: Filter,

Filters the output stream using a given Filter.

fn boxed(self) -> Box<dyn DynTokenizer>

where Self: Sized + 'static,

Creates a boxed version of the tokenizer.