Constant tantivy::tokenizer::MAX_TOKEN_LEN

pub const MAX_TOKEN_LEN: usize = _; // 65_530usize

Maximum authorized len (in bytes) for a token.

Tokenizers are in charge of not emitting tokens larger than this value. Currently, if a faulty tokenizer implementation emits tokens with a length larger than 2^16 - 1 - 5, the token will simply be ignored downstream.