bert_tokenizer 0.1.0

This crate is a Rust port of Google's BERT WordPiece tokenizer.
Documentation 
1
2
3
4
5
6
7
8
9

# bert_tokenizer

\[ [API doc](https://docs.rs/bert_tokenizer/) | [crates.io](https://crates.io/crates/bert_tokenizer/) \]

The crate provides the port of the original BERT tokenizer from the [Google BERT repository](https://github.com/google-research/bert).

## License

MIT license. See [LICENSE](LICENSE) file for full license.