nipah_tokenizer

A powerful yet simple text tokenizer for your everyday needs!

How To Use

1. Obtain the default options (or create yourself your configuration):

let options = nipah_tokenizer::options::default();

2. Get some text

let text = "Hello, World!".to_string();

3. Tokenize your text!

use nipah_tokenizer::tokenizer::tokenize;

let tokens = tokenize(text, &options);

Output will be like:

["Hello": Id] [Comma] ["World": Id] [Exclamation]

You can use it as your hearth desires, and it is pretty versatile

Happy coding!