Module llm 

LLM token metering (gateway L0).

When [llm] is enabled, EdgeGuard parses OpenAI-compatible traffic to meter tokens and cost — the substrate every later level (budgets, governance, cost accounting) builds on. L0 is metering only: it never blocks, rewrites, or delays a request. Token counts come from the upstream’s own usage object (authoritative), so the proxy does not tokenize anything itself.

Two response shapes are handled:

• non-streaming — a JSON body carrying usage.{prompt,completion}_tokens (parse_response_usage);
• streaming (SSE) — the terminal data: frame carries usage when the client sets stream_options.include_usage (parse_sse_usage); without it, no usage is emitted and the request is metered as no_usage.

Pricing is a per-model book (LlmRuntime). What happens to a request for an unmapped model is a config choice (UnpricedPolicy): count keeps the historical fail-open behaviour (tokens counted, cost omitted — surfaced as the unpriced result), while block fails closed (402, the request never reaches the upstream) so a mispriced/unknown model can’t be served at a silent $0. Cost is accumulated in micro-dollars (1e-6 USD) as an integer to avoid float drift in a monotonic counter.

Token accounting captures four dimensions, not two: prompt and completion, plus the cached prompt tokens (prompt_tokens_details.cached_tokens) and the reasoning completion tokens (completion_tokens_details.reasoning_tokens) that OpenAI-compatible providers bill differently. Pricing them separately is what fixes the “~7× undercount” on reasoning/cached traffic; when a model leaves the cached/reasoning rate unset they inherit the input/output rate, so a book that predates those knobs prices exactly as before.

Structs

LlmRuntime

The compiled LLM runtime: whether metering is on, the API style, and the price book. Built once per config (re)load and carried on the proxy Runtime.

Usage

Token usage as reported by the upstream’s usage object. cached_tokens is the subset of prompt_tokens served from the provider’s prompt cache; reasoning_tokens is the subset of completion_tokens spent on hidden reasoning. Both are carried separately so they can be priced (and metered) at their own rate rather than folded into the base input/output totals.

Enums

UnpricedPolicy

What to do with a request whose model is not in the price book.

Functions

canonical_model

The canonical model name for attribution (budgets, per-model rollups): the bare name with a known provider prefix stripped, else the name unchanged. So a per-model budget and its rollups aggregate "openai/gpt-4o" and "gpt-4o" as one model instead of splitting spend across two buckets (a prefixed request otherwise silently escaping a bare-named budget).

estimate_prompt_tokens

A rough prompt-token estimate from the raw request size (~4 bytes/token, the common English heuristic). Deliberately an over-estimate — the JSON envelope inflates it — so the budget reserve errs toward caution (a hard cap should never admit past the limit); the reservation is reconciled down to the upstream’s exact usage afterward.

parse_request_max_tokens

Extract the request’s completion-token ceiling (max_tokens, or max_completion_tokens). Used only to size the budget reserve estimate; the reservation is reconciled to actual usage after.

parse_request_model

Extract the model field from an OpenAI-style request body. None if the body isn’t JSON or has no model (then the request isn’t metered as LLM traffic). Other fields are ignored, so a large messages array is not materialized beyond what serde must scan.

parse_response_usage

Extract usage from a non-streaming OpenAI-style response body. None if absent/zero (an error response or a stream that didn’t include usage).

parse_sse_usage

Extract the terminal usage from an SSE stream’s bytes. OpenAI emits a final data: {…, "usage": {…}} frame when the client sets stream_options.include_usage; earlier frames carry "usage": null. Returns the last non-empty usage seen (the authoritative totals), or None if the stream never reported usage.