itok
Built by an LLM, deliberately and in the open
This repository — code, spec, tests and prose — was written by Claude Code running Anthropic's Claude Opus 5. 93 of 107 commits carry a
Co-Authored-By: Claude Opus 5trailer. A human owns every decision, reviews every diff, and is accountable for what ships.The method is spec-driven development.
SPEC.mdis the law rather than a description written afterwards: the invariants that must stay true, the tasks that remain, and every bug found so far paired with the rule that now catches it. A rule and its checker land in the same commit, because a rule with no runner gates nothing (§V17).The guardrails are git hooks that also run on CI. Entering the dev shell (
nix develop, ordirenv allow) installspre-commitandpre-push, which run hk against one definition of the gate inhk.pkl— 26 steps on commit, 32 on push, the slow half adding coverage, the tarball's contents and the network axis.ci.ymlcalls that same definition, so a laptop and a runner cannot disagree. The badges above are generated from those runs rather than typed here, because a number typed into prose is true the day it is written and quietly wrong after.The record is deliberately unflattering.
§Blogs what got through:B16records that the spec promised TLS the build could not perform, and that no test covered the path — found by a reader, not by the gate.Deeper:
AGENTS.mdis the working guide ·CONTRIBUTING.mdis the loop ·INTEGRATION.mdis the gate and its known gaps.
Estimate the token / context cost of files and changes, from the command
line. git- and du-shaped, so if you know those tools you already know
this one.
Honest by design: the default is an estimate and says so out loud.
itok never claims a measurement it cannot make.
$ itok estimate -h --top 3
~25k itok SPEC.md
~9k itok src/topcmd.rs
~8k itok pkl/Config.pkl
~43k itok total (bytes/4)
itok= input tokens -- what a file costs fed into a model (not output/generation).~marks a crude estimate. A real tokenizer count drops it.(bytes/4)names the method, so the number always says how it was reached.
Why
A context window is a budget, and most tools spend it silently. You discover what a file cost after it is already resident — re-billed on every turn, crowding out the reasoning you were trying to buy.
itok makes that cost visible before you pay it, with the grammar you
already use for the other resource you count: du for what is big, git diff for what changed. Point it at a tree and it tells you what entering
costs; point it at a commit and it tells you what the change added.
The design rule underneath is that a number must name its own method: ~
marks an estimate, a real tokenizer count drops it, an exact count names the
endpoint that produced it. A confident figure the tool cannot justify is the
defect this one exists to avoid.
Install
With Nix, no Rust toolchain needed:
Three outputs: default (the --bpe tokenizer included), itok-minimal
(the zero-dependency bytes/4 core), and itok-ollama (adds the
LAN-exact tier).
Or from a checkout:
That puts itok in ~/.cargo/bin -- make sure it is on your PATH.
Prefix inference
Verb names support prefix inference: any unambiguous prefix works, so
itok e, itok est, and itok estimate are the same command. A typo that
is not a prefix is suggested, never run:
The full command reference is below; itok docs prints it and
is the source of that section.
The precision ladder
An estimate is only as good as its method. itok offers a ladder from
cheap-and-crude to true, and always labels which rung produced a number.
flowchart LR
A["dummy<br/>bytes / 4<br/>zero-dep, instant"] --> B["--bpe<br/>real tokenizer<br/>offline, deterministic"] --> C["--ollama<br/>a local model's own<br/>tokenizer, exact, LAN"]
style A fill:#eee,stroke:#999
All three rungs are built. dummy (bytes/4, the ~4-chars-per-token rule
of thumb, off by roughly 15-30%) is the zero-dependency default; --bpe
is a real tokenizer (o200k, offline, deterministic, no tilde); --ollama
gets an exact count from a local model's own tokenizer over the LAN --
keyless, no cloud, no network in the core. Because every number names its
method, you always know which you are looking at.
Budgets
Supplying --budget N turns estimate (or diff) into a gate: it fails
if a file -- or a change's delta -- exceeds the budget. The report still
prints; the breach goes to stderr; the exit code is 1.
)
)
N accepts a decimal unit: 2000, 15k, 1M. Drop it into CI as a
one-line ceiling with no config file to maintain.
JSON output
--format json is a stable contract -- one JSON object per file, so
tools can parse it without guessing:
}
estimated is true for a crude tier and false for a real tokenizer --
the machine-readable form of the ~ marker.
Commands
Every verb, its synopsis and what it does. Regenerate with itok docs.
estimate
estimate [-s] [-h] [--top N] [--budget N] [--bpe] [--ollama[=HOSTS]] [--format human|json] [-C dir] [paths...]
Token cost of files, git-tracked by default. --bpe swaps bytes/4 for a real tokenizer (o200k); --ollama gets an exact count from a local model's own tokenizer; a bare host needs the = form (--ollama=$OLLAMA_HOST). --budget N turns it into a gate.
doctor
doctor [--model X[,Y...]] [--window N] [--ollama[=HOSTS]] [--format human|json] [-C dir] [paths...]
Advisory health check: fit-to-window, budget balance, noise ratio, estimate confidence. Reports and suggests; never gates. --model X resolves an encoding via .context-models, and --model a,b narrows an --ollama fleet to those models (one unresolvable name fails the call); --ollama[=HOSTS] discovers live model windows across a fleet; a bare host needs the = form.
diff
diff [<A> <B> | <A>..<B> | <ref>] [--staged] [--exit-code] [--budget N] [--bpe] [-- <path>]
Token delta between two points (default: working tree vs HEAD), git-diff-shaped. --exit-code or --budget N makes it a gate.
show
show [<commit>] [-- <path>] | show <commit>:<path>
One commit's per-file token delta (default HEAD). The <commit>:<path> form reports a single blob's cost at that ref.
log
log <path> [<A>..<B>] [-n N] [--since D] [--reverse] [--bpe] [--format human|json]
A path's token cost and delta across every commit that touched it -- the creep curve. Report-only, git-log-shaped.
check
check [-C dir] [--format human|json]
Gate registered paths against .context-limits (pinned --bpe, so the verdict is deterministic). Exit 1 on any breach.
guard
guard
Hook adapter for a harness: reads one hook payload on stdin, writes a decision on stdout, one process per call. Decides from .context-policy -- per-glob and per-tool budgets, with pins allowed absolutely. No policy file means allow, silently, so enforcement never self-enables. The decision is in the JSON, never in the exit code.
fit
fit --window N [--by size] [--bpe] [--format human|json] [-C dir] [paths...]
Greedy subset of files that fits a token window; emits a pipeable path list (git-tracked by default). itok fit --window 200k src/ | xargs cat builds a context bundle under budget.
trace
trace [<session>] [-n N] [--since D] [--reverse] [--format human|json]
Runtime load events for a session, one line each, chronologically -- what entered the context, when, and how big. Defaults to the newest transcript for the working directory. Report-only. Per-event sizes are estimates (bytes/4): no content is stored, so there is nothing to tokenize.
top
top [<session>] [-- <path>] [-h] [-s] [--top N] [--format human|json]
Ranked context occupancy for a session, du-shaped: how much each thing cost, how many times it was loaded, how many turns have passed since, and how much cache re-billing it has carried since it entered (size x turns remaining -- the number that makes early reduction's leverage visible; assumes no compaction). -- <path> narrows to one path's loads. Ends with the accounted-vs-unaccounted split: what itok can attribute against what the model actually received, each naming its method. Report-only; per-load sizes are estimates (bytes/4).
headroom
headroom [<session>] [--model X] [--window N] [--task N] [-h] [--format human|json] [-C dir]
df for a context: window, used, avail, use% -- plus the growth rate over the last 10/50/200 TURNS (context grows per turn, so a per-second rate would be meaningless) and ~turns left at the recent rate. Without --window or --model there is no capacity, so avail/use%/turns left are reported as absent rather than computed against a guessed window. --task N adds a tasks left column (avail/N) -- N is the WINDOW a task occupies, not what it bills. Report-only.
calibrate
calibrate [<session>] [-h] [--format human|json] [-C dir]
What this session's context actually cost, against what itok estimated: a fixed overhead the transcript cannot see (system prompt + tool schemas) and a scale from bytes/4 to real tokens. Reports the error BAND measured on turns the fit never saw, plus n -- never a bare factor. Too few turns reports n and no factor. The scale absorbs message framing and unrecorded reasoning, so it is not a tokenizer ratio and is derived per session. Report-only.
cap
cap [N] [--footer human|json]
Token-budget filter for a pipe: reads stdin, emits the longest whole-line prefix that fits N tokens, and ANNOUNCES the cut in a footer -- what was kept, what was elided, and the line and byte offset to resume from, so the next read continues rather than restarting (the line number is exact on any stream; the byte offset is of the decoded text, so it is for UTF-8 input). head truncates silently by lines or bytes; this truncates by tokens and says so. Without N nothing is cut and the footer just reports the cost. Report-only, exit 0.
docs
docs
Print this command reference as markdown -- the source for README's generated block, kept in sync by a guard.
Exit codes
| code | meaning |
|---|---|
0 |
ok |
1 |
budget breach or nonzero delta |
2 |
usage error |
7 |
network error (--ollama) |
The command reference above is generated: itok docs prints it, and a
test fails if this block drifts from the code. Edit the registry in
src/docs.rs, never the block by hand.
Use it as a library
The CLI is a thin shell over a pure function, so a caller gets the same verdict without spawning a process:
use cli;
let out = run;
assert_eq!;
Output carries stdout, stderr and the exit code the binary would have
used. The pieces are public too — estimate, bpe, session, walk,
render, json — so a consumer can take the measurement without the
grammar around it.
This is not hypothetical: blackbox
depends on itok with features = ["bpe"] to size the slices it feeds to a
model.
Guarantees
Each of these is checked by something, not asserted here:
- A number names its method. Every figure carries the tier that produced
it —
bytes/4,o200k, orexact via <model>@<host>— and~marks a crude estimate.itoknever claims a measurement it cannot make. - Deterministic. The same input yields the same count.
checkpins its tier so a gate's verdict cannot drift with a flag. - Offline by default. The default build needs no network at all. The exact tier is feature-gated and opt-in, and its tests replay a recorded cassette rather than reaching a server.
- The minimal tier is genuinely zero-dependency —
--no-default-featuresresolves to nothing, and that is a build in CI (nix build .#itok-minimal), not a promise in prose. - The JSON contract is stable. One object per file, so a caller parses rather than guesses.
- Reporting and gating are separate. Report-only verbs never fail;
gating is an explicit act —
--budget,check,guard— so nothing starts refusing your commits because you asked it a question. - No
unsafe.unsafe_code = "forbid", so it cannot be reintroduced in a local module.
Status
0.3.0 — the first public release. A minor here is a level of
guarantee, not a feature count, and 0.3 is an odd minor, which the
ladder reserves for a fix release between milestones (§V70). It says what is
not done: 0.4 means "knows what a context costs", and that is still open.
Publication is deliberately not a rung (§V107). The ladder once made
0.7 the first public release, welding who-can-install-it onto
what-it-guarantees — leaving only two moves, wait or overstate. This ships at
the rung that is true, early, because real use finds what another pass over
our own tree does not.
An -rc.1 came first because crates.io is immutable. It found five defects
no dry run reaches, two of which would have been permanent.
Fourteen verbs are built. The reduction ladder (cap --strip and friends)
and doctor --session are specced and not built — SPEC.md §I marks
them so, and the binary rejects them rather than pretending.
Changelog
See CHANGELOG.md.
Contributing
itok is built spec-first: SPEC.md is both the design and the build
queue, and a contribution means taking the next task from it and making it
pass the gate.
- docs/CONTRIBUTING.md — setup, the loop, and what gets a patch turned down
- docs/INTEGRATION.md — how the gate is wired, and
how to reproduce any verdict without
hk - docs/CODE_OF_CONDUCT.md — how discussion here is expected to go
- AGENTS.md — what to do when a specific step fails
Security
Report privately rather than in a public issue — see docs/SECURITY.md, which also states plainly what the attack surface is and is not.
If you are reporting something involving session transcripts, do not attach a real one: they contain actual conversation content.
License
MIT. See LICENSE.
Third-party dependencies vary by feature tier — the minimal build has none at all. See docs/THIRD-PARTY-NOTICES.md.