Expand description
itok – estimate the token / context cost of files and changes, from
the command line. git- and du-shaped, honest by design (the default
is a labelled estimate, never a claim of truth).
This crate is primarily the itok binary; the library exists so the
whole command surface is unit- and property-tested at lib cost (main.rs
is a thin shell). See the README for usage and SPEC.md for the design
invariants.
Modules§
- bpe
- Real-tokenizer tier (V4):
o200k_basevia tiktoken-rs. The ranks are embedded in the crate, so this is offline at runtime and deterministic (V12, V5); only the build-time crate fetch touches the network, and the flake vendors that. Behind thebpefeature, so--no-default-featuresyields the zero-dep dummy-only tool (V13). - cli
- The CLI as a pure function: args in,
Output(streams + exit code) out. No I/O here –mainprints the strings and sets the code. This is what makes the whole command surface unit- and property-testable at lib cost, instead of spawning the binary: V245-style totality (“never panics on any input”) becomes a proptest, not a hope. - estimate
- Dummy tier + file selection into estimates. The dummy tier is ~4 chars per token, the standard rule of thumb: zero-dep, instant (V4). The word proxy joins at doctor (T13), where the dummy-vs-bpe spread becomes the confidence signal; estimate reports one honest number and always names its method (V3).
- glob
- Path patterns for
.context-policy(V57). - json
- The stable machine contract: one JSON object per file (JSONL), the
plumbing half of the porcelain/plumbing split (V9). Hand-written –
zero deps, no serde (V13). Fields carry V3’s intent:
unitis input_tokens,estimatedflags the crude tier (the json form of the human tilde). Numbers stay numeric; no tilde ever appears here. - render
- Human rendering – the cosmetic table, free to evolve. The stable
machine contract (
--format json) is a separate concern (V9, T3). - session
- The runtime axis’s reader (V41): what a session actually LOADED, from the harness’s own on-disk transcript.
- walk
- File selection: git-tracked by default (V8), like
rgrespects.gitignore. Reimplemented here rather than imported from the host’srepo-guard– zero host deps (V13), so this survives extraction.