logreduce-tokenizer

Build:

python setup.py build
export PYTHONPATH=$(pwd)/build/lib

Bench:

python benches/bench.py

Test perf:

RUSTFLAGS="-C target-cpu=native" cargo build --release

Build CLI:

RUSTFLAGS="-C target-cpu=native" cargo build --example logreduce-tokenizer-cli --release