mtc-inc-bpe 0.9.0

Incremental BPE tokenization for all prefixes
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# mtc-inc-bpe

[![Crates.io](https://img.shields.io/crates/v/mtc-inc-bpe.svg)](https://crates.io/crates/mtc-inc-bpe)
[![Docs.rs](https://img.shields.io/docsrs/mtc-inc-bpe.svg)](https://docs.rs/mtc-inc-bpe)
[![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-informational.svg)](#license)
[![Build status](https://github.com/ModelTC/mtc-inc-bpe/actions/workflows/ci.yml/badge.svg)](https://github.com/ModelTC/mtc-inc-bpe/actions)

Incremental BPE tokenization for all prefixes.

## License

- &copy; 2025 Chielo Newctle \<[ChieloNewctle@gmail.com](mailto:ChieloNewctle@gmail.com)\>
- &copy; 2025 ModelTC Team

This project is licensed under either of

- [MIT license](https://opensource.org/licenses/MIT) ([`LICENSE-MIT`](LICENSE-MIT))
- [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0) ([`LICENSE-APACHE`](LICENSE-APACHE))

at your option.

The [SPDX](https://spdx.dev) license identifier for this project is `MIT OR Apache-2.0`.