TokenGeeX - Efficient Tokenizer for CodeGeeX
This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for CodeGeeX aimed at code and Chinese. It is based on UnigramLM (Taku Kudo 2018).
- Coverage
- 32.73%
18 out of 55 items documented0 out of 11 items with examples - Size
- Source code size: 208.29 kB This is the summed size of all the files inside the crates.io package for this release.
- Documentation size: 11.67 MB This is the summed size of all files generated by rustdoc for all configured targets
- Links
- Homepage
-
dhbrojas/tokengeex
4 2 2 - crates.io
- Dependencies
-
- base64 ^0.21.7 normal
- dashmap ^5.5.3 normal
- derive_builder ^0.12 normal
- env_logger ^0.9 normal
- fancy-regex ^0.13.0 normal
- fnv ^1.0.7 normal
- log ^0.4 normal
- memmap2 ^0.5.2 normal
- rand ^0.8.5 normal
- rayon ^1.8 normal
- rayon-cond ^0.3 normal
- regex ^1.10.3 normal
- serde ^1.0.196 normal
- serde_json ^1.0.113 normal
- thiserror ^1.0.57 normal
- unicode-normalization ^0.1.23 normal
- unicode_categories ^0.1.1 normal
- xflags ^0.3.2 normal
- assert_approx_eq ^1.1 dev
- criterion ^0.3 dev
- Versions
- Owners
-