Constant tantivy::tokenizer::MAX_TOKEN_LEN [−][src]
pub const MAX_TOKEN_LEN: usize = u16::max_value() as usize - 4; // 0x0_000_000_000_00f_ffbusize
Maximum authorized len (in bytes) for a token.
Tokenizer are in charge of not emitting tokens larger than this value.
Currently, if a faulty tokenizer implementation emits tokens with a length larger than
2^16 - 1 - 4
, the token will simply be ignored downstream.