Module tokenizer 

Tokenizer implementation for morphological analysis.

This module provides a builder pattern for creating tokenizers and the tokenizer itself. The build-flow orchestration is delegated to lindera_binding_core::CoreTokenizerBuilder / lindera_binding_core::CoreTokenizer; this module only adds the PyO3 wrappers and the PyDict-to-JSON conversion.

Examples

# Create a tokenizer with custom configuration
tokenizer = (lindera.TokenizerBuilder()
    .set_mode("normal")
    .append_token_filter("japanese_stop_tags", {"tags": ["助詞"]})
    .build())

# Tokenize text
tokens = tokenizer.tokenize("すもももももももものうち")

Structs

PyTokenizer

Tokenizer for performing morphological analysis.

PyTokenizerBuilder

Builder for creating a Tokenizer with custom configuration.

Functions

register