Module tokenizer 

Shared tokenizer build-flow orchestration for the bindings.

Each binding’s tokenizer wrapper reimplements the same flow: configure a lindera::tokenizer::TokenizerBuilder, build a lindera::tokenizer::Tokenizer, and convert the resulting tokens. This module collects that orchestration into CoreTokenizerBuilder and CoreTokenizer, leaving each binding to do only its FFI-value conversion (serde_json::Value ⇔ the host language’s argument type) and a thin wrapper.

Structs

CoreTokenizer

Tokenizer that orchestrates tokenization on behalf of the bindings.

CoreTokenizerBuilder

Builder that orchestrates tokenizer configuration on behalf of the bindings.