smoltoken-0.2.0
- smoltoken 0.2.0
- Permalink
- Docs.rs crate page
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- svarunid
- Dependencies
- base64 ^0.22 normal
  fancy-regex ^0.14.0 normal
  kdam ^0.6.2 normal
  rayon ^1.10.0 normal
  rustc-hash ^2.0.0 normal
  walkdir ^2 normal
- Versions
- 95.24% of the crate is documented
Platform
Feature flags

docs.rs

Rust

All

smoltoken0.2.0

Crate Items

Structs
Enums

List of all items

Structs

BytePairTokenizer
FakeThreadId

Enums

DecodeError
TokenizerDataSource