Tocken

Tokenizer implemented in Rust.

This tokenizer is based on Lucene's EnglishAnalyzer.

Usage

• as a library: check the main.rs file and docs.
• as a CLI:
  • cargo r -r --help
  • cargo r -r -- -i wiki.txt -o wiki_tocken_f10.json -f 10