dynamo-llm-0.2.0
- dynamo-llm 0.2.0
- Docs.rs crate page
- Apache-2.0
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- nv-anants
- saturley-hall
- dmitry-tokarev-nv
- nv-nmailhot
- Dependencies
- akin ^0.4.0 normal
  
  anyhow ^1 normal
  
  async-nats ^0.40 normal
  
  async-openai ^0.27.2 normal
  
  async-stream ^0.3 normal
  
  async-trait ^0.1 normal
  
  async_zmq ^0.4.0 normal
  
  axum ^0.8 normal
  
  blake3 ^1 normal
  
  bs62 ^0.1 normal
  
  bytemuck ^1.22 normal
  
  bytes ^1 normal
  
  candle-core ^0.8.0 normal
  
  chrono ^0.4 normal
  
  cudarc ^0.16.2 normal optional
  
  derive-getters ^0.5 normal
  
  derive_builder ^0.20 normal
  
  dynamo-runtime ^0.2.0 normal
  
  either ^1.13 normal
  
  erased-serde ^0.4 normal
  
  futures ^0.3 normal
  
  galil-seiferas ^0.1 normal
  
  ggus ^0.4.0 normal
  
  itertools ^0.14.0 normal
  
  memmap2 ^0.9.5 normal
  
  minijinja ^2.3.1 normal
  
  minijinja-contrib ^2.3.1 normal
  
  ndarray ^0.16 normal optional
  
  prometheus ^0.14 normal
  
  rand ^0.9.0 normal
  
  rayon ^1 normal
  
  regex ^1 normal
  
  sentencepiece ^0.11.2 normal optional
  
  serde ^1 normal
  
  serde_json ^1 normal
  
  strum ^0.27 normal
  
  thiserror ^2.0.11 normal
  
  tokenizers ^0.21.1 normal
  
  tokio ^1 normal
  
  tokio-stream ^0.1 normal
  
  tokio-util ^0.7 normal
  
  toktrie ^0.6.28 normal
  
  toktrie_hf_tokenizers ^0.6.28 normal
  
  tracing ^0.1 normal
  
  unicode-segmentation ^1.12 normal
  
  url ^2.5 normal
  
  uuid ^1 normal
  
  validator ^0.20.0 normal
  
  xxhash-rust ^0.8 normal
  
  hf-hub ^0.4.2 dev
  
  insta ^1.41 dev
  
  proptest ^1.5.0 dev
  
  reqwest ^0.12 dev
  
  rstest ^0.18.2 dev
  
  rstest_reuse ^0.7.0 dev
  
  tempfile ^3.17.1 dev
- Versions
- 43.68% of the crate is documented
Go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

dynamo_llm0.2.0

Tokenizer

Implementors

In dynamo_llm::tokenizers::traits

dynamo_llm::tokenizers::traits

Trait Tokenizer

pub trait Tokenizer: Encoder + Decoder { }

Implementors§

impl Tokenizer for HuggingFaceTokenizer