Skip to main content

Module effort

Module effort 

Source
Expand description

Cache-safe, cross-provider reasoning-effort control (#834).

Operators pin a single reasoning-effort level (proxy.effort) that lean-ctx translates to each provider’s native parameter. Unlike per-turn “effort routing” — which changes the effort between turns of one conversation and thereby invalidates the provider prompt cache (OpenAI lists “changes to reasoning effort” as a cache-invalidation cause; Anthropic breaks message cache breakpoints on thinking-mode/config changes) — this value is a constant: identical on every request, so the cached prefix stays byte-stable (#448/#498) and only the model’s reasoning depth changes.

Safety rules, enforced by every applier:

  • Opt-in: the caller only invokes an applier when proxy.effort is set; off is a strict no-op that preserves the byte-unchanged meter-only path.
  • Never override the client: an effort the client set explicitly is left untouched, so the request keeps the client’s own cache key.
  • Never enable reasoning the client didn’t ask for: the Anthropic applier only dials an existing adaptive request, so it never adds thinking tokens (or a 400) where the client wanted none. OpenAI reasoning models always reason, so setting the level only ever caps/redirects existing reasoning. The Gemini applier excludes 2.5 flash-lite (thinking off by default) for the same reason, and never sends both thinkingLevel and thinkingBudget.
  • Model-gated: models that would reject the parameter are skipped, so the feature can never turn a working request into a 400. Gemini’s generation is read from the URL path (thinkingLevel on 3.x, thinkingBudget on 2.5).
  • Deterministic: the rewrite is a pure function of (document, level), so identical requests stay byte-identical across turns.

Structs§

EffortStats
Point-in-time view of the effort control for /status: the active level (so an operator can confirm proxy.effort is live) plus how many requests have been steered per provider. Pair the counters with the per-model reasoning_tokens the usage meter already records to see the realized output-token savings.

Functions§

apply_anthropic
Set output_config.effort on an Anthropic request, but only when the client already requested adaptive thinking (thinking.type == "adaptive"). That guard means lean-ctx never enables thinking the client didn’t ask for (no surprise reasoning cost) and never sends adaptive config to a model that rejects it (the client already proved the model supports it). No-op when the client already set output_config.effort.
apply_google
Set the Gemini thinking control that matches model’s generation on a generateContent request (thinkingLevel for 3.x, thinkingBudget for 2.5). model comes from the request URL path — Gemini carries it there, not in the body — threaded in by the Google handler. No-op when the model is unknown/excluded, when the client already pinned either thinking field (never override, and never end up with both → 400), or on an unexpected body shape.
apply_openai_chat
Set reasoning_effort on an OpenAI Chat Completions request. No-op when the client already set it or the model is non-reasoning. Returns whether the document changed.
apply_openai_responses
Set reasoning.effort on an OpenAI Responses request (nested object). No-op when the client already pinned reasoning.effort or the model is non-reasoning. Any other reasoning.* fields (e.g. summary) are preserved.
openai_supports_effort
Whether an OpenAI model accepts a reasoning-effort parameter. Reasoning models (the o-series and the gpt-5/gpt-6 families, including any codex build) do; the non-reasoning gpt-4*/gpt-3* models and the *-chat-latest non-reasoning variants reject it with a 400, so they are excluded. A vendor-prefixed name from an OpenAI-compatible gateway (openai/gpt-5.5, openrouter/openai/o3) is reduced to its bare model segment first.
snapshot
Snapshot the counters together with the currently resolved effort level (None"off").