Struct InMemoryRateLimiter 

pub struct InMemoryRateLimiter { /* private fields */ }

An in memory rate limiter based on a token bucket algorithm.

This is an in memory rate limiter, so it cannot rate limit across different processes.

The rate limiter only allows time-based rate limiting and does not take into account any information about the input or the output, so it cannot be used to rate limit based on the size of the request.

It is thread safe and can be used in either a sync or async context.

The in memory rate limiter is based on a token bucket. The bucket is filled with tokens at a given rate. Each request consumes a token. If there are not enough tokens in the bucket, the request is blocked until there are enough tokens.

These tokens have nothing to do with LLM tokens. They are just a way to keep track of how many requests can be made at a given time.

Current limitations:

• The rate limiter is not designed to work across different processes. It is an in-memory rate limiter, but it is thread safe.
• The rate limiter only supports time-based rate limiting. It does not take into account the size of the request or any other factors.

Example

use agent_chain_core::rate_limiters::{InMemoryRateLimiter, InMemoryRateLimiterConfig, BaseRateLimiter};

let rate_limiter = InMemoryRateLimiter::new(InMemoryRateLimiterConfig {
    requests_per_second: 0.1,  // Can only make a request once every 10 seconds
    check_every_n_seconds: 0.1,  // Wake up every 100 ms to check whether allowed to make a request
    max_bucket_size: 10.0,  // Controls the maximum burst size
});

// In sync context
rate_limiter.acquire(true);

// In async context
rate_limiter.aacquire(true).await;

Implementations

impl InMemoryRateLimiter

pub fn new(config: InMemoryRateLimiterConfig) -> Self

Create a new InMemoryRateLimiter with the given configuration.

These tokens have nothing to do with LLM tokens. They are just a way to keep track of how many requests can be made at a given time.

This rate limiter is designed to work in a threaded environment.

It works by filling up a bucket with tokens at a given rate. Each request consumes a given number of tokens. If there are not enough tokens in the bucket, the request is blocked until there are enough tokens.

Trait Implementations

impl BaseRateLimiter for InMemoryRateLimiter

fn acquire(&self, blocking: bool) -> bool

Attempt to acquire a token from the rate limiter.

This method blocks until the required tokens are available if blocking is set to true.

If blocking is set to false, the method will immediately return the result of the attempt to acquire the tokens.

Arguments

• blocking - If true, the method will block until the tokens are available. If false, the method will return immediately with the result of the attempt.

Returns

true if the tokens were successfully acquired, false otherwise.

fn aacquire<'life0, 'async_trait>( &'life0 self, blocking: bool, ) -> Pin<Box<dyn Future<Output = bool> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Attempt to acquire a token from the rate limiter. Async version.

This method blocks until the required tokens are available if blocking is set to true.

If blocking is set to false, the method will immediately return the result of the attempt to acquire the tokens.

Arguments

• blocking - If true, the method will block until the tokens are available. If false, the method will return immediately with the result of the attempt.

Returns

true if the tokens were successfully acquired, false otherwise.