itok 0.3.2

context-cost estimator: token and window budgets for files and changes
Documentation

itok

Read LLM-DISCLAIMER first. This repository is tended by an autonomous loop, and that file says what the loop may do here, what it may not, and what to check before trusting anything in this tree.

CI License: MIT crates.io docs.rs edition 2024 MSRV 1.95 direct dependencies 5 minimal tier 0 dependencies unsafe forbidden gate hk coverage 98.3% floor 98%

nix flake nixpkgs da6dc0e intel linux intel macos amd linux amd macos arm linux arm macos

built with Claude Code built with Opus 5 built with SDD

Read LLM-DISCLAIMER first.

Estimate the token / context cost of files and changes, from the command line. git- and du-shaped, so if you know those tools you already know this one.

Honest by design: the default is an estimate and says so out loud. itok never claims a measurement it cannot make.

$ itok estimate -h --top 3
       ~25k itok  SPEC.md
        ~9k itok  src/topcmd.rs
        ~8k itok  pkl/Config.pkl
       ~43k itok  total (bytes/4)
  • itok = input tokens -- what a file costs fed into a model (not output/generation).
  • ~ marks a crude estimate. A real tokenizer count drops it.
  • (bytes/4) names the method, so the number always says how it was reached.

Why

A context window is a budget, and most tools spend it silently. You discover what a file cost after it is already resident — re-billed on every turn, crowding out the reasoning you were trying to buy.

itok makes that cost visible before you pay it, with the grammar you already use for the other resource you count: du for what is big, git diff for what changed. Point it at a tree and it tells you what entering costs; point it at a commit and it tells you what the change added.

The design rule underneath is that a number must name its own method: ~ marks an estimate, a real tokenizer count drops it, an exact count names the endpoint that produced it. A confident figure the tool cannot justify is the defect this one exists to avoid.

Install

cargo install itok

With Nix, no Rust toolchain needed:

nix run github:pr0d1r2/itok -- estimate SPEC.md   # run without installing
nix profile install github:pr0d1r2/itok           # or install

Three outputs: default (the --bpe tokenizer included), itok-minimal (the zero-dependency bytes/4 core), and itok-ollama (adds the LAN-exact tier).

Or from a checkout:

cargo install --path .

That puts itok in ~/.cargo/bin -- make sure it is on your PATH.

Prefix inference

Verb names support prefix inference: any unambiguous prefix works, so itok e, itok est, and itok estimate are the same command. A typo that is not a prefix is suggested, never run:

$ itok esimate
itok: unknown command 'esimate' -- did you mean estimate?

The full command reference is below; itok docs prints it and is the source of that section.

The precision ladder

An estimate is only as good as its method. itok offers a ladder from cheap-and-crude to true, and always labels which rung produced a number.

flowchart LR
    A["dummy<br/>bytes / 4<br/>zero-dep, instant"] --> B["--bpe<br/>real tokenizer<br/>offline, deterministic"] --> C["--ollama<br/>a local model's own<br/>tokenizer, exact, LAN"]
    style A fill:#eee,stroke:#999

All three rungs are built. dummy (bytes/4, the ~4-chars-per-token rule of thumb, off by roughly 15-30%) is the zero-dependency default; --bpe is a real tokenizer (o200k, offline, deterministic, no tilde); --ollama gets an exact count from a local model's own tokenizer over the LAN -- keyless, no cloud, no network in the core. Because every number names its method, you always know which you are looking at.

Budgets

Supplying --budget N turns estimate (or diff) into a gate: it fails if a file -- or a change's delta -- exceeds the budget. The report still prints; the breach goes to stderr; the exit code is 1.

$ itok estimate --budget 20k SPEC.md
   ~38k itok  SPEC.md
   ~38k itok  total (bytes/4)
itok: 1 file(s) over budget 20000:
  38785 itok  SPEC.md
$ echo $?
1

N accepts a decimal unit: 2000, 15k, 1M. Drop it into CI as a one-line ceiling with no config file to maintain.

JSON output

--format json is a stable contract -- one JSON object per file, so tools can parse it without guessing:

$ itok estimate --format json SPEC.md
{"path":"SPEC.md","tokens":38785,"unit":"input_tokens","estimated":true,"method":"bytes/4"}

estimated is true for a crude tier and false for a real tokenizer -- the machine-readable form of the ~ marker.

Commands

Every verb, its synopsis and what it does. Regenerate with itok docs.

estimate

estimate [-s] [-h] [--top N] [--budget N] [--bpe] [--ollama[=HOSTS]] [--format human|json] [-C dir] [paths...]

Token cost of files, git-tracked by default. --budget N turns it into a gate.

--bpe swaps bytes/4 for a real tokenizer (o200k); --ollama gets an exact count from a local model's own tokenizer, and a bare host needs the = form (--ollama=$OLLAMA_HOST). The two tiers agree on DELTAS even where they differ on absolutes, so --bpe answers which of two texts costs less without a fleet.

doctor

doctor [--session [<id>]] [--model X[,Y...]] [--window N] [--ollama[=HOSTS]] [-h] [--format human|json] [-C dir] [paths...]

Advisory health check over a fileset, or over a running session with --session. Reports and suggests; never gates.

Over a fileset: fit-to-window, budget balance, noise ratio, estimate confidence. --model X resolves an encoding via .context-models; --model a,b narrows an --ollama fleet, and one unresolvable name fails the call.

--session [<id>] retargets the whole verb at a running context. It prints headroom's row -- window, used, avail, use%, the rate triple, ~turns left -- then projects each item's occupancy across the turns remaining, biggest first. A transcript carrying a compaction boundary already dropped items, so its projection is labelled an upper bound. No capacity means no ~turns left and no projection, and the report says which absence it is. A second positional is a usage error: this form has no fileset.

When use% crosses the line fit warns at, and only then, it ends with advice: the two levers measured to help, and the one it rejects by name.

diff

diff [<A> <B> | <A>..<B> | <ref>] [--staged] [--exit-code] [--budget N] [--bpe] [-- <path>]

Token delta between two points, git-diff-shaped. Default is the working tree against HEAD.

--exit-code or --budget N makes it a gate.

show

show [<commit>] [-- <path>] | show <commit>:<path>

One commit's per-file token delta. Default HEAD.

The <commit>:<path> form reports a single blob's cost at that ref.

log

log <path> [<A>..<B>] [-n N] [--since D] [--reverse] [--bpe] [--format human|json]

A path's token cost and delta across every commit that touched it -- the creep curve.

Report-only, git-log-shaped.

check

check [-C dir] [--format human|json]

Gate registered paths against .context-limits. Exit 1 on any breach.

The tokenizer is pinned (--bpe), so the verdict is deterministic across machines.

guard

guard

Hook adapter: one harness payload on stdin, one decision on stdout.

Decides from .context-policy -- per-glob and per-tool budgets, with pins allowed absolutely. No policy file means allow, silently, so enforcement never self-enables. The decision is in the JSON, never in the exit code.

fit

fit --window N [--by size] [--bpe] [--format human|json] [-C dir] [paths...]

Greedy subset of files that fits a token window; emits a pipeable path list.

Git-tracked by default. itok fit --window 200k src/ | xargs cat builds a context bundle under budget.

trace

trace [<session>] [-n N] [--since D] [--reverse] [--format human|json]

Runtime load events for a session, one line each, chronologically.

What entered the context, when, and how big. Defaults to the newest transcript for the working directory. Report-only, and per-event sizes are estimates (bytes/4): no content is stored, so there is nothing to tokenize.

top

top [<session>] [-- <path>] [-h] [-s] [--top N] [--cost --model X] [--format human|json] [-C dir]

Ranked context occupancy for a session, du-shaped. Report-only.

How much each thing cost, how many times it was loaded, how many turns since, and how much cache re-billing it has carried since it entered. Whether that product is exact is read off the session: a transcript carrying a compaction boundary had items leave, so its carried is an upper bound and the report names when.

-- <path> narrows to one path's loads. Ends with the accounted-vs-unaccounted split, each number naming its method.

--cost --model X prices the session, using the in/cr/cc column in .context-models -- three integers per million tokens, in whatever unit that file declares. itok bundles no price list and names no currency: prices change and vary by contract, so a built-in one would go stale into a confident lie. The rates multiply the harness's own usage split, not this report's rows, because a row's tokens are an estimate and its cache split is unattributable. A model with no rate column, or no --model at all, prints why there is no price rather than printing none.

headroom

headroom [<session>] [--model X] [--window N] [--task N] [-h] [--format human|json] [-C dir]

df for a context: window, used, avail, use%, the growth rate, and ~turns left. Report-only.

The rate is over the last 10/50/200 TURNS -- context grows per turn, so a per-second rate would be meaningless. Without --window or --model there is no capacity, so avail/use%/turns left report as absent rather than against a guessed window. --task N adds a tasks left column: N is the WINDOW a task occupies, not what it bills.

calibrate

calibrate [<session>] [-h] [--format human|json] [-C dir]

What a session's context actually cost against what itok estimated.

A fixed overhead the transcript cannot see (system prompt + tool schemas) and a scale from bytes/4 to real tokens, with the error BAND measured on turns the fit never saw, plus n -- never a bare factor. Too few turns reports n and no factor. The scale absorbs message framing and unrecorded reasoning, so it is derived per session and is not a tokenizer ratio.

rate

rate [<session>] [--statusline] [--color auto|always|never] [--format human|json] [-C dir]

Pre-formatted badge for a statusline: occupancy, the bill, and growth per hour and per day.

Cell 1 is a LEVEL: the last turn's billed input, coloured as a fill gauge against the point the session is heading for -- [rate].compact if declared (one number, or a table keyed by model), else the auto-compact point this session's own transcript recorded, else the model's window. 24-bit where COLORTERM allows, three bands otherwise, and the last tenth also takes a !. A zero window is never painted: an absent measurement is not an empty context.

Cells 3 and 4 divide GROWTH -- the sum of positive window deltas -- not the bill, which is mostly cache re-reads and so measures how long the session has run rather than how full it is getting. Cell 2 keeps the bill under a name that says so; it is what the API was asked to charge, which is neither money nor content.

A fifth cell appears when a red line resolved and the context is growing: time to it at the recent rate, floored and tilde'd, in ACTIVE seconds. Rates divide by active time, each inter-turn gap credited up to 300 seconds.

--statusline reads the harness payload on stdin and emits (itok:...); colour defaults to always there. --color reads [rate] thresholds from itok.toml; a metric with no threshold gets no colour. 0-1 turns hides the badge, but --format json still returns one object with null where nothing was measured.

cap

cap [N] [--strip] [--dedup] [--elide] [--outline] [--footer human|json]

Token-budget filter for a pipe: the longest whole-line prefix that fits N tokens, and it announces the cut.

The footer says what was kept, what was elided, and the line and byte offset to resume from, so the next read continues rather than restarting. The line number is exact on any stream; the byte offset is of the decoded text, so it is for UTF-8 input. head truncates silently by lines or bytes; this truncates by tokens and says so. Without N nothing is cut and the footer just reports the cost.

The reduction ladder is four opt-in rungs, ordered by what they cost a reader: --strip (ANSI escapes, trailing whitespace), --dedup (runs of identical lines collapse, keeping the count), --elide (long dense base64 and minified lines replaced by their size) and --outline (indented bodies dropped, signature-shaped lines kept). They run in LADDER order whatever order you pass them, stopping at the first rung that meets the budget, and the footer names every rung that ran. Order is the invariant, not your flag order: letting the caller put structural loss ahead of whitespace removal is the failure the ladder exists to prevent.

A rung above cap rewrites the text, so line and byte offsets no longer index your stream: the resume selector is absent and the footer says why. The input totals stay your own bytes, so the elision is measured against what you handed in rather than against what the ladder left.

docs

docs

Print this command reference as markdown -- the source for README's generated block.

Kept in sync by a guard, so the block and this table cannot disagree.

Exit codes

code meaning
0 ok
1 budget breach or nonzero delta
2 usage error
7 network error (--ollama)

The command reference above is generated: itok docs prints it, and a test fails if this block drifts from the code. Edit the registry in src/docs.rs, never the block by hand.

Use it as a library

The CLI is a thin shell over a pure function, so a caller gets the same verdict without spawning a process:

use itok::cli;
let out = cli::run(&["estimate".into(), "--format".into(), "json".into()]);
assert_eq!(out.code, 0);

Output carries stdout, stderr and the exit code the binary would have used. The pieces are public too — estimate, bpe, session, walk, render, json, glob — so a consumer can take the measurement without the grammar around it. glob::matches is the three wildcards only, not gitignore's rules (*.md does not match docs/guide.md), and estimate::select_paths is the tracked-by-default selection estimate itself uses; both document their contract.

This is not hypothetical: sherd depends on itok with default-features = false, features = ["bpe"] to size the slices it feeds to a model.

Guarantees

Each of these is checked by something, not asserted here:

  • A number names its method. Every figure carries the tier that produced it — bytes/4, o200k, or exact via <model>@<host> — and ~ marks a crude estimate. itok never claims a measurement it cannot make.
  • Deterministic. The same input yields the same count. check pins its tier so a gate's verdict cannot drift with a flag.
  • Offline by default. The default build needs no network at all. The exact tier is feature-gated and opt-in, and its tests replay a recorded cassette rather than reaching a server.
  • The minimal tier is genuinely zero-dependency — --no-default-features resolves to nothing, and that is a build in CI (nix build .#itok-minimal), not a promise in prose.
  • The JSON contract is stable. One object per file, so a caller parses rather than guesses.
  • Reporting and gating are separate. Report-only verbs never fail; gating is an explicit act — --budget, check, guard — so nothing starts refusing your commits because you asked it a question.
  • No unsafe. unsafe_code = "forbid", so it cannot be reintroduced in a local module.

Status

0.3.0 — the first public release. A minor here is a level of guarantee, not a feature count, and 0.3 is an odd minor, which the ladder reserves for a fix release between milestones (§V70). It says what is not done: 0.4 means "knows what a context costs", and that is still open.

Publication is deliberately not a rung (§V107). The ladder once made 0.7 the first public release, welding who-can-install-it onto what-it-guarantees — leaving only two moves, wait or overstate. This ships at the rung that is true, early, because real use finds what another pass over our own tree does not.

An -rc.1 came first because crates.io is immutable. It found five defects no dry run reaches, two of which would have been permanent.

Fourteen verbs are built. The reduction ladder (cap --strip and friends) and doctor --session are specced and not built — SPEC.md §I marks them so, and the binary rejects them rather than pretending.

Changelog

See CHANGELOG.md.

Contributing

itok is built spec-first: SPEC.md is both the design and the build queue, and a contribution means taking the next task from it and making it pass the gate.

Security

Report privately rather than in a public issue — see docs/SECURITY.md, which also states plainly what the attack surface is and is not.

If you are reporting something involving session transcripts, do not attach a real one: they contain actual conversation content.

License

MIT. See LICENSE.

Third-party dependencies vary by feature tier — the minimal build has none at all. See docs/THIRD-PARTY-NOTICES.md.