Expand description
Tokenizer implementation for morphological analysis.
This module provides a builder pattern for creating tokenizers and the tokenizer itself.
The build-flow orchestration is delegated to
lindera_binding_core::CoreTokenizerBuilder / lindera_binding_core::CoreTokenizer;
this module only adds the napi wrappers and the JS-value conversion.
Structsยง
- JsTokenizer
- Tokenizer for performing morphological analysis.
- JsTokenizer
Builder - Builder for creating a Tokenizer with custom configuration.