Module tokens 

The shared token estimator (SPEC.md C9). The repo has no tokenizer — only provider-reported Usage (provider.rs:153-164) and the turn/output counters agent.rs already tracks. Every other UX figure (the banner, /status, /tokens, show-reductions, notices) derives from one documented heuristic here, and is always printed with a ~ prefix so it reads as an estimate, never a measurement. Acceptance criteria never assert against these numbers directly — ACs assert exact byte counts (SessionInfo::full_bytes/view_bytes, C9/D15) and only derive a token/dollar figure from them for the test log.

Provider-reported figures (Usage.completion_tokens, B7’s optional cached_tokens, agent.total_output_tokens()) are real counts and are never routed through this module — they print un-tilded.

Constants

CONTEXT_RESPONSE_RESERVE_TOKENS

PARITY-18 — headroom reserved for the model’s own completion, folded into the context-guard boundary alongside with_guard_margin. Neither estimate_view_tokens nor estimate_request_tokens counts anything for the reply the model is about to generate — this is a flat token budget carved out of the model’s context window for it, since the completion shares the same window as the request on every provider this crate targets.

Functions

context_guard

PARITY-18 D1/D2/D4 — the single context-guard decision, shared by every call site that must decide whether a request is safe to send: the CLI’s resume_cmd preflight check AND Agent::run_loop’s per-send check (D4 — the guard is a session invariant, not a one-shot preflight, so turn 2+ and /expand all are covered too). Because both call through this one function, a request can never pass one gate and fail the other — there is only one formula.

estimate_deferred_schema_tokens

Schema-token estimate for the B6 “tools” banner line: the estimate over the serialized full ToolSchema list minus the estimate over the serialized advertised list — i.e. the token cost of what’s currently deferred (hidden behind tool_search) rather than eagerly advertised. Saturates to 0 rather than underflow if advertised somehow estimates larger than full (e.g. formatting differences), since “negative deferred tokens” has no meaning for the banner.

estimate_request_tokens

PARITY-18 D1 — the full wire-request token estimate: every message in messages (callers pass what crate::Agent::build_request_messages actually returns, which already carries the system prompt at index 0) PLUS the serialized tools schema array, which is a real part of the request body ([crate::provider::build_request_body]) but — before PARITY-18’s re-fix — was never counted by the preflight guard at all. A session whose messages alone fit comfortably could still carry a fat builtin/MCP tool-schema array that blows the real wire request; this is the fix.

estimate_tokens

Deterministic token estimate: ceil(utf8_bytes / 4). Documented heuristic — all UX figures derived from it are printed with a ~ prefix (see fmt_approx_tokens). Never used in acceptance criteria (ACs assert exact byte counts).

estimate_view_tokens

Estimate for a serialized message list: the wire form (serde_json) used to build the provider request, not any in-memory/debug representation.

fmt_approx_tokens

Render an estimated token count in the shared UX style: ~21,904 tok (tilde prefix + comma-grouped thousands, matching the stub-line comma style in reduce.rs). Every figure that flows through estimate_tokens/estimate_view_tokens/estimate_deferred_schema_tokens should be rendered through this helper so the ~ discipline (D11) is applied uniformly rather than ad hoc at each call site.

with_guard_margin

Apply the [GUARD_MARGIN_NUM]/[GUARD_MARGIN_DEN] context-guard safety margin to a raw token estimate. Rounds up (div_ceil), never down — the margin only ever pushes the boundary check to be more cautious.