ferrum-tokenizer-0.7.0
- ferrum-tokenizer 0.7.0
- Docs.rs crate page
- MIT
- Links
- crates.io
- Source
- Owners
- sizzlecar
- Dependencies
- anyhow ^1.0 normal
  async-trait ^0.1 normal
  dashmap ^6.1 normal
  dirs ^5.0 normal
  ferrum-interfaces ^0.7.0 normal
  ferrum-types ^0.7.0 normal
  futures ^0.3 normal
  hf-hub ^0.4.1 normal
  once_cell ^1.19 normal
  parking_lot ^0.12 normal
  regex ^1.10 normal optional
  serde ^1.0 normal
  serde_json ^1.0 normal
  thiserror ^1.0 normal
  tokenizers ^0.21 normal
  tokio ^1.35 normal
  tracing ^0.1 normal
  uuid ^1.6 normal
  tokio-test ^0.4 dev
- Versions
- 84.62% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

Files

ferrum_tokenizer/implementations/
mod.rs

1pub mod huggingface;
2
3pub use huggingface::{HuggingFaceTokenizer, HuggingFaceTokenizerFactory, IncrementalState};