Module cooldown 

Per-agent and per-model cooldown tracking.

Shared by engine::runner::response (recording failures) and engine::router::config (avoiding cooled models during pool selection).

Persistence

All cooldowns are persisted to the KV store in SQLite so they survive service restarts. Call init_cooldown_store once at startup (after the TaskStore is open) to register the store and pre-load unexpired cooldowns.

When a rate limit includes a “try again at {date}” message, the cooldown is set to that specific timestamp instead of the default duration. Relative usage windows (e.g. “5-hour usage limit”) are also detected and used to set a model-specific cooldown matching the advertised window.

Structs

SilenceCountResult

Enums

CreditExhaustionReason

Credit exhaustion reason detected from error message.

Constants

AGENT_COOLDOWN_SECS

Default fallback cooldown for agents when no store is available: 30 minutes.

BACKOFF_BASE_SECS

Base backoff for generic agent/model failures: 5 minutes.

BACKOFF_MAX_SECS

Maximum backoff for generic agent/model failures: 4 hours.

BILLING_CYCLE_COOLDOWN_SECS

Base cooldown for billing cycle exhaustion: 24 hours.

BILLING_CYCLE_MAX_SECS

Maximum cooldown for billing cycle exhaustion: 7 days.

CREDIT_BACKOFF_BASE_SECS

Base backoff for credit exhaustion (out_of_credits): 1 hour.

CREDIT_BACKOFF_MAX_SECS

Standard cap for credit exhaustion and org-level disabling: 8 hours. Repeated failures at the cap move to the extended tiers (48h, then 96h)

DEFAULT_DEGRADED_RATE_LIMIT_THRESHOLD

Default threshold: minimum rate-limit events within the window to consider an agent degraded. A single transient 429 shouldn’t mark an agent; we need a pattern of repeated failures.

DEFAULT_HEALTH_CHECK_WINDOW_HOURS

Default health check window for recent rate-limit events (hours).

ORG_BACKOFF_BASE_SECS

Base backoff for org-level disabling: 2 hours.

PERSISTENT_MODEL_BACKOFF_BASE_SECS

Base backoff for persistent model failures (silent exit/provider errors): 4 hours.

PERSISTENT_MODEL_BACKOFF_MAX_SECS

Maximum backoff for persistent model failures: 7 days.

SILENCE_AGENT_COOLDOWN_SECS

Short agent cooldown applied on silence detection (120 seconds).

SILENCE_COUNT_THRESHOLD

Silence detections before applying an extended cooldown (rolling window).

SILENCE_COUNT_WINDOW_SECS

Rolling window for silence detections (24 hours).

SILENCE_EXTENDED_COOLDOWN_SECS

Extended cooldown applied after repeated silence detections (4 hours).

Functions

clear_agent_degraded

Clear the degraded flag for an agent.

clear_cooldown

Clear a specific cooldown by key, or all cooldowns when key == "*".

clear_expired_cooldowns

Clear expired cooldowns from the in-memory map.

compute_backoff

Compute exponential backoff duration using base-3 growth.

cooldown_reason

Return the reason string recorded when a cooldown was set, if the cooldown is still active.

cooldown_until

Return the cooldown expiry timestamp for a key (agent or agent:model), if active.

detect_credit_exhaustion

Detect credit exhaustion reasons from error message.

detect_credit_exhaustion_outcome

Detect credit exhaustion from an error message and return the stable task_runs.outcome string for it (see CreditExhaustionReason::outcome_str).

github_circuit_remaining_secs

Returns the remaining seconds of the GitHub 5xx circuit breaker cooldown, or 0 if the circuit is closed.

init_cooldown_store

is_agent_degraded

Check if an agent is currently flagged as degraded by the health check.

is_agent_in_cooldown

Check if an agent is currently in cooldown period.

is_github_circuit_open

Check if the GitHub 5xx circuit breaker is currently open.

is_model_in_cooldown

Check if a specific agent+model combo is in cooldown.

list_all_cooldowns

Return all currently active cooldowns as (key, cooldown_until_unix, reason).

mark_agent_degraded

Mark an agent as degraded (called by refresh_degraded_agents).

record_agent_failure_with_message

Record that an agent has failed and should be temporarily avoided.

record_agent_success

Reset failure counts for an agent and agent:model combo after a successful run.

record_credit_exhaustion

Record an agent-level cooldown for credit exhaustion.

record_github_5xx

Record a GitHub 5xx server error and trip the circuit breaker if the sliding window threshold is exceeded.

record_model_failure

Record that a specific agent+model combo has failed.

record_persistent_model_failure

Record a persistent model-specific failure and apply longer exponential backoff.

record_rate_limit

Record a rate-limit event for an agent+model.

record_silence_detection

Record a silence detection for an agent+model and apply extended cooldowns when repeated silences exceed the threshold within the rolling window.

refresh_degraded_agents

Refresh the degraded-agent set from the rate_limits table and cooldown state.

set_agent_cooldown

Set a short agent-level cooldown (in seconds).

set_model_cooldown

Set a model cooldown with a custom duration (in seconds) and reason.

set_model_cooldown_until

Set a model cooldown that ends at a vendor-provided Unix timestamp.

sync_from_kv

Reconcile the in-memory cooldown map with the persisted KV state.