Module cshannon::tokens

Modules

bytes	bytes module implements tokenization of a string into bytes.
graphemes	graphemes module implements tokenization of a string into unicode grapheme clusters.
test_utils
words	words module implements tokenization of a string into unicode words.

Traits

Token	A single item in the tokenized stream from a string input.
TokenIter	An iterator for Tokens read from a Reader.
TokenPacker

Functions

pack_all	Packs a vector of tokens prefixed with the length of the vector.
unpack_all	Unpacks a vector of tokens previously packed with pack_with_len().
unpack_u64