Module tokenizer 

Enums

SegmentLanguage

The language a Tokenizer::Segment segments with.

Tokenizer

Functions

write_tokenizers_sql

Writes tokenizer keywords separated by commas (no spaces), as in TOKENIZERS BLANK,CAMEL.