tiktoken

English | 简体中文 | 日本語

The fastest Rust BPE tokenizer — 15–40x faster than tiktoken-rs on ASCII text (≈2x on CJK/Unicode), thanks to a hand-written ASCII fast-path. Compatible with OpenAI tiktoken and supports all mainstream LLM tokenizers — OpenAI, Llama 3, DeepSeek, Qwen, and Mistral.

Features

• Multi-provider: 17 encodings across 8 vendors (OpenAI, Meta, DeepSeek, Alibaba, Mistral, Moonshot, Zhipu, MiniMax)
• Fast: hand-written ASCII fast-path pre-tokenizer (bypasses the regex), arena-based vocabulary, hybrid BPE merge
• Parallel encoding: optional rayon-based multi-threaded encoding for large texts
• Pricing: cost estimation for 107 models across 10 providers
• Compact: ruzstd-compressed vocabulary data embedded at compile time
• Zero-alloc counting: count() path avoids token vector allocation

Performance

Token outputs are asserted identical across implementations before anything is timed; each figure is one full pass over the corpus, median of 9 rounds after warmup. Corpora are byte-identical in every harness (bench-compare/, benches/bench_python.py, web/bench/).

Native — Apple M4 Mac mini, single thread, encode

cargo run --release -p bench-compare

o200k_base tracks the same ratios (5–48x vs tiktoken-rs). count() runs another 5–15% faster than encode — it never allocates the id vector.

In the browser — Mac Studio (M4 Max), Chromium

npm run bench in web/ — this crate compiled to wasm, against the two mainstream JavaScript tokenizers.

The adversarial corpus never repeats a piece, which disables every implementation's memoisation — it is the floor, and the lead holds there too.

The scanners are pinned to the regex by property tests (hundreds of thousands of random inputs per run), and every encoding is pinned to its vendor's tokenizer by 44,518 differential fixture cases.

Benchmark source: benches/, ../bench-compare/. Reproducible via cargo bench / cargo run -p bench-compare.

Installation

[dependencies]
tiktoken = "3"

# optional: multi-threaded encoding for large texts
tiktoken = { version = "3", features = ["parallel"] }

Quick Start

// by encoding name
let enc = tiktoken::get_encoding("cl100k_base").unwrap();
let tokens = enc.encode("hello world");
let text = enc.decode_to_string(&tokens).unwrap();
assert_eq!(text, "hello world");

// by model name — works across all providers
let enc = tiktoken::encoding_for_model("gpt-4o").unwrap();
let count = enc.count("hello world"); // zero-alloc fast path

let enc = tiktoken::encoding_for_model("llama-3.1-70b").unwrap();
let enc = tiktoken::encoding_for_model("deepseek-v3").unwrap();
let enc = tiktoken::encoding_for_model("qwen2.5-72b").unwrap();

Supported Encodings

API

Encode / decode

let enc = tiktoken::get_encoding("cl100k_base").unwrap();

let tokens = enc.encode("hello world");           // Vec<u32>
let text = enc.decode_to_string(&tokens).unwrap(); // String
let bytes = enc.decode(&tokens);                   // Vec<u8>

Special tokens

let enc = tiktoken::get_encoding("cl100k_base").unwrap();
let tokens = enc.encode_with_special_tokens("hello<|endoftext|>world");
// tokens will include the special token id for <|endoftext|>

Count tokens

let enc = tiktoken::get_encoding("o200k_base").unwrap();
let count = enc.count("The quick brown fox jumps over the lazy dog.");
// faster than encode().len() -- no token vector allocated

// count with special token recognition
let count = enc.count_with_special_tokens("hello<|endoftext|>world");

Parallel encoding

// requires `parallel` feature
let enc = tiktoken::get_encoding("cl100k_base").unwrap();
let tokens = enc.encode_parallel("...very long text...");
// identical output, uses rayon for texts >= 4KB

Cost estimation

use tiktoken::pricing;

// quick estimate
let cost = pricing::estimate_cost("gpt-4o", 1_000_000, 500_000).unwrap();

// with prompt caching
let model = pricing::get_model("claude-opus-4").unwrap();
let cost = model.estimate_cost_with_cache(500_000, 500_000, 200_000);

// list all models for a provider
let models = pricing::models_by_provider(pricing::Provider::DeepSeek);

Supports 107 models across OpenAI, Anthropic, Google, Meta, DeepSeek, Alibaba, and Mistral.

WebAssembly

Available as @goliapkg/tiktoken-wasm on npm.

npm install @goliapkg/tiktoken-wasm

import init, { getEncoding, encodingForModel, estimateCost } from '@goliapkg/tiktoken-wasm'

await init()

const enc = getEncoding('cl100k_base')
const tokens = enc.encode('hello world')    // Uint32Array
const text = enc.decode(tokens)             // "hello world"
const count = enc.count('hello world')      // 2

enc.free() // release WASM memory when done

See examples/react-app for a complete demo.

# requires wasm-pack: cargo install wasm-pack
cd tiktoken-wasm
wasm-pack build --target web --release

Ecosystem

tiktoken · @goliapkg/tiktoken-wasm · instructors · chunkedrs · embedrs

License

Licensed under either of MIT or Apache-2.0, at your option.

Third-party vocabulary data licenses: see LICENSE-3RD-PARTY.