vibrato 0.5.2

Vibrato: viterbi-based accelerated tokenizer
Documentation
DEFAULT	       0 1 0
SPACE	       0 1 0
KANJI	       0 0 2
SYMBOL	       1 1 0
NUMERIC	       1 1 0
ALPHA	       1 1 0
HIRAGANA       0 1 2
KATAKANA       1 1 2
KANJINUMERIC   1 1 0
GREEK	       1 1 0
CYRILLIC       1 1 0

0x0020 SPACE

0x0030..0x0039 NUMERIC
0xFF10..0xFF19 NUMERIC

0x0061..0x007A ALPHA
0xFF21..0xFF3A ALPHA
0xFF41..0xFF5A ALPHA

0x3040..0x3096 HIRAGANA

0x30A0..0x30FA KATAKANA
0x30FC..0x30FF KATAKANA
0xFF66..0xFF9F KATAKANA

0x3400..0x4DBF KANJI
0x4E00..0x9FFF KANJI
0xF900..0xFAFF KANJI

0x4E00 KANJINUMERIC KANJI
0x4E8C KANJINUMERIC KANJI