Struct RetryMiddleware 

pub struct RetryMiddleware { /* private fields */ }

Middleware that retries failed calls with exponential backoff.

Retry policy:

• Only errors with ProviderError::is_retryable are retried; everything else propagates immediately.
• Self::wrap_generate: retries the full request up to max_attempts times.
• Self::wrap_stream: retries opening the stream only. Once the stream is open (any item, including crate::language_model::StreamPart::Error, has been delivered), the retry policy stops; callers decide whether to re-issue the call.

Backoff is deterministic (no jitter); add jitter at the caller level if you have hundreds of concurrent retriers hitting the same upstream.

Examples

use std::sync::Arc;
use std::time::Duration;
use llmsdk_provider::{wrap_language_model, LanguageModel, LanguageModelMiddleware};
use llmsdk_provider::middleware::RetryMiddleware;

fn add_retry(model: Arc<dyn LanguageModel>) -> Arc<dyn LanguageModel> {
    let retry = RetryMiddleware::builder()
        .max_attempts(5)
        .initial_backoff(Duration::from_millis(200))
        .build();
    wrap_language_model(model, [Arc::new(retry) as Arc<dyn LanguageModelMiddleware>])
}

Implementations

impl RetryMiddleware

pub fn new() -> Self

Build with default policy (DEFAULT_MAX_ATTEMPTS, 100ms initial, x2 multiplier, 5s cap).

pub fn builder() -> RetryMiddlewareBuilder

Open a builder for non-default tuning.

Trait Implementations

impl Clone for RetryMiddleware

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RetryMiddleware

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RetryMiddleware

fn default() -> Self

Returns the “default value” for a type.

impl LanguageModelMiddleware for RetryMiddleware

fn wrap_generate<'life0, 'life1, 'async_trait>( &'life0 self, next: &'life1 dyn LanguageModel, params: CallOptions, ) -> Pin<Box<dyn Future<Output = Result<GenerateResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Wrap a non-streaming generation.

fn wrap_stream<'life0, 'life1, 'async_trait>( &'life0 self, next: &'life1 dyn LanguageModel, params: CallOptions, ) -> Pin<Box<dyn Future<Output = Result<StreamResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Wrap a streaming generation.

fn override_provider(&self, _inner: &dyn LanguageModel) -> Option<String>

Override the provider id exposed by the wrapped model.

fn override_model_id(&self, _inner: &dyn LanguageModel) -> Option<String>

Override the model id exposed by the wrapped model.

fn override_supported_urls<'life0, 'life1, 'async_trait>( &'life0 self, _inner: &'life1 dyn LanguageModel, ) -> Pin<Box<dyn Future<Output = Option<SupportedUrls>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Override the supported-URL map exposed by the wrapped model.

fn transform_params<'life0, 'life1, 'async_trait>( &'life0 self, _kind: CallKind, params: CallOptions, _inner: &'life1 dyn LanguageModel, ) -> Pin<Box<dyn Future<Output = Result<CallOptions>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Transform the call options before they reach the inner model.