tokengeex-1.1.0
- tokengeex 1.1.0
- Permalink
- Docs.rs crate page
- Apache-2.0
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- rojas-diego
- Dependencies
- base64 ^0.21.7 normal
  dashmap ^5.5.3 normal
  derive_builder ^0.12 normal
  env_logger ^0.9 normal
  fancy-regex ^0.13.0 normal
  fnv ^1.0.7 normal
  log ^0.4 normal
  memmap2 ^0.5.2 normal
  rand ^0.8.5 normal
  rayon ^1.8 normal
  rayon-cond ^0.3 normal
  regex ^1.10.3 normal
  serde ^1.0.196 normal
  serde_json ^1.0.113 normal
  thiserror ^1.0.57 normal
  unicode-normalization ^0.1.23 normal
  unicode_categories ^0.1.1 normal
  xflags ^0.3.2 normal
  assert_approx_eq ^1.1 dev
  criterion ^0.3 dev
- Versions
- 32.73% of the crate is documented
Platform
Feature flags

docs.rs

Rust

All

tokengeex1.1.0

Crate Items

Structs
Enums
Traits
Functions
Type Aliases

List of all items

Structs

CrlfProcessor
Lattice
LocalTask
Model
Node
ScoredToken
Task
Tokenizer
Trie
TrieIterator
VecPool

Enums

Error
ProcessorWrapper
UnicodeProcessor

Traits

Processor

Functions

make_vocab
mb_per_sec
new_default_vocab
par_chunk_size

Type Aliases

Result
Token
TokenID