Module tokenizer::en

A whitespace based word tokenizer.

It split each word by using white space, nothing else is done.

Structs

Tokenizer

White space based tokenizer. It split word based on white space.