vil_tokenizer 0.3.0

VIL Tokenizer Engine — native Rust BPE tokenization for LLM token counting and text splitting
Documentation 
1
2
3
4
5
6
7
8
9
10
11

# vil_tokenizer

VIL Tokenizer Engine — native Rust BPE tokenization for LLM token counting and text splitting

## Part of VIL

This crate is part of [VIL](https://github.com/OceanOS-id/VIL) — a process-oriented language and framework for building zero-copy, high-performance distributed systems.

## License

Licensed under either of [Apache License 2.0](../../LICENSE-APACHE) or [MIT License](../../LICENSE-MIT).