This is a tokenizer crate for the morsels_indexer cli tool, not intended to be used directly.
morsels_indexer