Struct RateLimit 

pub struct RateLimit { /* private fields */ }

Middleware that calls RateLimiter::acquire before each LLM call.

Implementations

impl RateLimit

pub fn new(limiter: Arc<dyn RateLimiter>) -> Self

Build with a limiter and the default char-count estimator.

pub fn with_estimator<F>(self, f: F) -> Self

where F: Fn(&MiddlewareCtx) -> u64 + Send + Sync + 'static,

Override the cost estimator (e.g. plug in a real tokenizer).

Trait Implementations

impl Middleware for RateLimit

fn call<'life0, 'async_trait>( &'life0 self, ctx: MiddlewareCtx, next: Arc<dyn Next>, ) -> Pin<Box<dyn Future<Output = Result<ChatResponse>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Wrap a call. Implementations typically:

fn name(&self) -> &str

Friendly name for telemetry / debugging.