ferrum-tokenizer-0.4.0
- ferrum-tokenizer 0.4.0
- Docs.rs crate page
- MIT
- Links
- crates.io
- Source
- Owners
- sizzlecar
- Dependencies
- anyhow ^1.0 normal
  async-trait ^0.1 normal
  dashmap ^6.1 normal
  dirs ^5.0 normal
  ferrum-interfaces ^0.4.0 normal
  ferrum-types ^0.4.0 normal
  futures ^0.3 normal
  hf-hub ^0.4.1 normal
  once_cell ^1.19 normal
  parking_lot ^0.12 normal
  regex ^1.10 normal optional
  serde ^1.0 normal
  serde_json ^1.0 normal
  thiserror ^1.0 normal
  tokenizers ^0.21 normal
  tokio ^1.35 normal
  tracing ^0.1 normal
  uuid ^1.6 normal
  tokio-test ^0.4 dev
- Versions
- 84.62% of the crate is documented
Go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

All

ferrum_tokenizer0.4.0

Crate Items

Structs
Enums
Traits
Functions
Type Aliases

List of all items

Structs

SpecialTokens
TokenId
TokenizerInfo
implementations::huggingface::HuggingFaceTokenizer
implementations::huggingface::HuggingFaceTokenizerFactory
implementations::huggingface::IncrementalState

Enums

TokenizerType

Traits

IncrementalTokenizer
Tokenizer
TokenizerFactory

Functions

default_factory
load_from_file
load_from_hub

Type Aliases

Result