Module tokenizer 

Tokenizer implementation for morphological analysis.

This module provides a builder pattern for creating tokenizers and the tokenizer itself.

Structs

RbTokenizer

Tokenizer for performing morphological analysis.

RbTokenizerBuilder

Builder for creating a Tokenizer with custom configuration.

Functions

define

Defines TokenizerBuilder and Tokenizer classes in the given Ruby module.