gproxy-tokenize 2.5.1

Offline token counting for LLM request bodies: tiktoken, Hugging Face vocabularies, character estimate
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
# Third-party notices

## DeepSeek V4 Pro tokenizer

- Asset: `assets/tokenizers/deepseek-v4-pro.tokenizer.json`
- Upstream: `deepseek-ai/DeepSeek-V4-Pro`
- Source revision: `b5968e9190ef611bbf34a7229255be88a0e937c1`
- SHA-256: `8f9f37ca37fdc4f5fd36d5cf4d3b0e8392edb4e894fd10cc0d70b4957c8633cf`
- License: MIT (the upstream license applies to this redistributed tokenizer)

The MIT license text is included in this crate's [`LICENSE`](LICENSE) file.
DeepSeek and its model/tokenizer names are trademarks of their respective
owners; inclusion does not imply endorsement.