Skip to main content

LanguageModelMiddleware

llmsdk_provider::middleware

Trait LanguageModelMiddleware 

Source 
pub trait LanguageModelMiddleware:
    Send
    + Sync
    + Debug {
    // Provided methods
    fn override_provider(&self, _inner: &dyn LanguageModel) -> Option<String> { ... }
    fn override_model_id(&self, _inner: &dyn LanguageModel) -> Option<String> { ... }
    fn override_supported_urls<'life0, 'life1, 'async_trait>(
        &'life0 self,
        _inner: &'life1 dyn LanguageModel,
    ) -> Pin<Box<dyn Future<Output = Option<SupportedUrls>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait { ... }
    fn transform_params<'life0, 'life1, 'async_trait>(
        &'life0 self,
        _kind: CallKind,
        params: CallOptions,
        _inner: &'life1 dyn LanguageModel,
    ) -> Pin<Box<dyn Future<Output = Result<CallOptions>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait { ... }
    fn wrap_generate<'life0, 'life1, 'async_trait>(
        &'life0 self,
        next: &'life1 dyn LanguageModel,
        params: CallOptions,
    ) -> Pin<Box<dyn Future<Output = Result<GenerateResult>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait { ... }
    fn wrap_stream<'life0, 'life1, 'async_trait>(
        &'life0 self,
        next: &'life1 dyn LanguageModel,
        params: CallOptions,
    ) -> Pin<Box<dyn Future<Output = Result<StreamResult>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait { ... }
}
Expand description

Contract for middleware that decorates a LanguageModel.

Every method has a sensible default, so an implementor only overrides the hooks it cares about. The combinator wrap_language_model composes any number of middlewares into a fresh LanguageModel instance.

§Semantics

  • override_*: replace the corresponding identity / metadata accessor.
  • transform_params: mutate the call options before they reach the underlying model. Runs once per call, before wrap_*.
  • wrap_generate / wrap_stream: intercept the actual call. The default implementation simply forwards to next; overrides may add retry, caching, instrumentation, swap between generate/stream, etc.

next is the next layer (which may itself be a wrapped model or the original provider model), not necessarily the underlying provider model.

Provided Methods§

Source

fn override_provider(&self, _inner: &dyn LanguageModel) -> Option<String>

Override the provider id exposed by the wrapped model.

Return None to keep inner.provider(). The override is read once when wrap_language_model runs, so it must not depend on call-time state.

Source

fn override_model_id(&self, _inner: &dyn LanguageModel) -> Option<String>

Override the model id exposed by the wrapped model.

Same caching semantics as Self::override_provider.

Source

fn override_supported_urls<'life0, 'life1, 'async_trait>( &'life0 self, _inner: &'life1 dyn LanguageModel, ) -> Pin<Box<dyn Future<Output = Option<SupportedUrls>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Override the supported-URL map exposed by the wrapped model.

Unlike the identity overrides, this is re-evaluated on every LanguageModel::supported_urls call so middleware can reflect dynamic provider state.

Source

fn transform_params<'life0, 'life1, 'async_trait>( &'life0 self, _kind: CallKind, params: CallOptions, _inner: &'life1 dyn LanguageModel, ) -> Pin<Box<dyn Future<Output = Result<CallOptions>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Transform the call options before they reach the inner model.

Runs once per call, before Self::wrap_generate / Self::wrap_stream. The returned options are passed to both the next middleware layer’s transform_params and the eventual underlying call.

§Errors

Return a crate::ProviderError to fail the call without invoking the model.

Source

fn wrap_generate<'life0, 'life1, 'async_trait>( &'life0 self, next: &'life1 dyn LanguageModel, params: CallOptions, ) -> Pin<Box<dyn Future<Output = Result<GenerateResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Wrap a non-streaming generation.

Default: forwards to next.do_generate(params). Override to add retry, caching, telemetry, or to dispatch to next.do_stream instead.

§Errors

Returns whatever error next returns, or a middleware-introduced failure.

Source

fn wrap_stream<'life0, 'life1, 'async_trait>( &'life0 self, next: &'life1 dyn LanguageModel, params: CallOptions, ) -> Pin<Box<dyn Future<Output = Result<StreamResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Wrap a streaming generation.

Default: forwards to next.do_stream(params). Override to add retry, caching, telemetry, or to simulate streaming on top of next.do_generate.

§Errors

Returns whatever error next returns, or a middleware-introduced failure.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§