Module pricing_cache 

Server-side cache of the LiteLLM pricing database.

On server startup we spawn a background task that fetches the LiteLLM JSON from GitHub and swaps it into an ArcSwap-style RwLock. If the fetch fails (offline, rate-limit, upstream 5xx) the cache stays at whatever it was — PricingDatabase::empty() initially, meaning the Claude fallback table in otelite-core::pricing handles every lookup until the first successful refresh.

The background task then sleeps for REFRESH_INTERVAL and tries again. Failures are logged but never fatal.

Structs

PricingCache

Thread-safe handle to the current pricing database.

PricingState

Constants

INITIAL_FETCH_TIMEOUT

Initial fetch timeout — if GitHub is slow we’d rather start the server with an empty cache and refresh in the background than block startup.

REFRESH_FETCH_TIMEOUT

Ongoing refresh timeout can be more generous.

REFRESH_INTERVAL

Refresh the pricing database once per day. LiteLLM updates the JSON when new models are added or prices change, typically weekly, so daily is ample.