gemini-tokenizer-0.2.0
- gemini-tokenizer 0.2.0
- Permalink
- Docs.rs crate page
- Apache-2.0
- Links
- crates.io
- Source
- Owners
- 5ocworkshop
- Dependencies
- sentencepiece ^0.11 normal
  serde ^1 normal
  serde_json ^1 normal
  sha2 ^0.10 normal
  serde_json ^1 dev
- Versions
- 97.5% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

VOCAB_SIZE

gemini_tokenizer0.2.0

gemini_tokenizer

Constant VOCAB_SIZE

pub const VOCAB_SIZE: usize = 262_144;

Expand description

The expected vocabulary size of the Gemma 3 tokenizer.