Struct CacheMiddleware 

pub struct CacheMiddleware { /* private fields */ }

Middleware that memoizes generate / stream responses keyed by CallOptions.

Combine with retry / logging via super::wrap_language_model; the recommended order is [logging, retry, cache, model] so cache hits skip retries entirely and logging records both fresh and cached calls.

Implementations

impl CacheMiddleware

pub fn new(store: Arc<dyn CacheStore>) -> Self

Build a middleware backed by store.

Trait Implementations

impl Clone for CacheMiddleware

fn clone(&self) -> CacheMiddleware

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CacheMiddleware

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl LanguageModelMiddleware for CacheMiddleware

fn wrap_generate<'life0, 'life1, 'async_trait>( &'life0 self, next: &'life1 dyn LanguageModel, params: CallOptions, ) -> Pin<Box<dyn Future<Output = Result<GenerateResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Wrap a non-streaming generation.

fn wrap_stream<'life0, 'life1, 'async_trait>( &'life0 self, next: &'life1 dyn LanguageModel, params: CallOptions, ) -> Pin<Box<dyn Future<Output = Result<StreamResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Wrap a streaming generation.

fn override_provider(&self, _inner: &dyn LanguageModel) -> Option<String>

Override the provider id exposed by the wrapped model.

fn override_model_id(&self, _inner: &dyn LanguageModel) -> Option<String>

Override the model id exposed by the wrapped model.

fn override_supported_urls<'life0, 'life1, 'async_trait>( &'life0 self, _inner: &'life1 dyn LanguageModel, ) -> Pin<Box<dyn Future<Output = Option<SupportedUrls>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Override the supported-URL map exposed by the wrapped model.

fn transform_params<'life0, 'life1, 'async_trait>( &'life0 self, _kind: CallKind, params: CallOptions, _inner: &'life1 dyn LanguageModel, ) -> Pin<Box<dyn Future<Output = Result<CallOptions>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Transform the call options before they reach the inner model.