Module tokenizer 

Tokenizer implementation for morphological analysis.

This module provides a builder pattern for creating tokenizers and the tokenizer itself. The build-flow orchestration is delegated to lindera_binding_core::CoreTokenizerBuilder / lindera_binding_core::CoreTokenizer; this module only adds the magnus wrappers and the Ruby-hash conversion.

Structs

RbTokenizer

Tokenizer for performing morphological analysis.

RbTokenizerBuilder

Builder for creating a Tokenizer with custom configuration.

Functions

define

Defines TokenizerBuilder and Tokenizer classes in the given Ruby module.