ferrum-tokenizer-0.7.0
- ferrum-tokenizer 0.7.0
- Docs.rs crate page
- MIT
- Links
- crates.io
- Source
- Owners
- sizzlecar
- Dependencies
- anyhow ^1.0 normal
  async-trait ^0.1 normal
  dashmap ^6.1 normal
  dirs ^5.0 normal
  ferrum-interfaces ^0.7.0 normal
  ferrum-types ^0.7.0 normal
  futures ^0.3 normal
  hf-hub ^0.4.1 normal
  once_cell ^1.19 normal
  parking_lot ^0.12 normal
  regex ^1.10 normal optional
  serde ^1.0 normal
  serde_json ^1.0 normal
  thiserror ^1.0 normal
  tokenizers ^0.21 normal
  tokio ^1.35 normal
  tracing ^0.1 normal
  uuid ^1.6 normal
  tokio-test ^0.4 dev
- Versions
- 84.62% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

load_from_file

ferrum_tokenizer0.7.0

ferrum_tokenizer

Function load_from_file

pub async fn load_from_file(path: &str) -> Result<Box<dyn Tokenizer>>

Expand description

Load tokenizer from file