Module cache_controller 

Prompt Caching Controller Module

Provides prompt caching support for reducing API costs and latency. This module implements cache control markers for eligible message blocks and calculates cache cost savings.

Features

• Add cache control markers to eligible messages
• Check cache eligibility based on token thresholds
• Calculate cache cost savings
• Track cache hit rates

Pricing Model

Based on Anthropic’s prompt caching pricing:

• Cache write: 1.25x base input price
• Cache read: 0.1x base input price (90% discount)

Structs

CacheController

Prompt Caching Controller

CacheEligibility

Result of cache eligibility check with indices of cacheable messages