Skip to main content

Crate itok

Crate itok 

Source
Expand description

itok – estimate the token / context cost of files and changes, from the command line. git- and du-shaped, honest by design (the default is a labelled estimate, never a claim of truth).

This crate is primarily the itok binary; the library exists so the whole command surface is unit- and property-tested at lib cost (main.rs is a thin shell). See the README for usage and SPEC.md for the design invariants.

Modules§

bpe
Real-tokenizer tier (V4): o200k_base via tiktoken-rs. The ranks are embedded in the crate, so this is offline at runtime and deterministic (V12, V5); only the build-time crate fetch touches the network, and the flake vendors that. Behind the bpe feature, so --no-default-features yields the zero-dep dummy-only tool (V13).
cli
The CLI as a pure function: args in, Output (streams + exit code) out. No I/O here – main prints the strings and sets the code. This is what makes the whole command surface unit- and property-testable at lib cost, instead of spawning the binary: V245-style totality (“never panics on any input”) becomes a proptest, not a hope.
estimate
Dummy tier + file selection into estimates. The dummy tier is ~4 chars per token, the standard rule of thumb: zero-dep, instant (V4). The word proxy joins at doctor (T13), where the dummy-vs-bpe spread becomes the confidence signal; estimate reports one honest number and always names its method (V3).
glob
Path patterns for .context-policy (V57).
json
The stable machine contract: one JSON object per file (JSONL), the plumbing half of the porcelain/plumbing split (V9). Hand-written – zero deps, no serde (V13). Fields carry V3’s intent: unit is input_tokens, estimated flags the crude tier (the json form of the human tilde). Numbers stay numeric; no tilde ever appears here.
render
Human rendering – the cosmetic table, free to evolve. The stable machine contract (--format json) is a separate concern (V9, T3).
session
The runtime axis’s reader (V41): what a session actually LOADED, from the harness’s own on-disk transcript.
walk
File selection: git-tracked by default (V8), like rg respects .gitignore. Reimplemented here rather than imported from the host’s repo-guard – zero host deps (V13), so this survives extraction.