Struct ConcurrencyController 

pub struct ConcurrencyController { /* private fields */ }

Concurrency controller for LLM API calls.

Combines:

• Rate Limiter — Token bucket to limit requests per time period
• Semaphore — Limit concurrent requests

Example

use vectorless::throttle::{ConcurrencyController, ConcurrencyConfig};

let config = ConcurrencyConfig::default();
let controller = ConcurrencyController::new(config);

// Before making an API call
let permit = controller.acquire().await;

// Make the API call...
drop(permit); // Release when done

Implementations

impl ConcurrencyController

pub fn new(config: ConcurrencyConfig) -> Self

Create a new concurrency controller with the given configuration.

pub fn with_defaults() -> Self

Create a controller with default configuration.

pub fn high_throughput() -> Self

Create a controller for high-throughput scenarios.

pub fn conservative() -> Self

Create a controller for conservative scenarios.

pub fn unlimited() -> Self

Create a controller with no limits.

pub async fn acquire(&self) -> Option<SemaphorePermit<'_>>

Acquire a permit for making an LLM request.

This will:

1. Wait for the rate limiter (if enabled)
2. Acquire a semaphore permit (if enabled)

The permit is automatically released when dropped.

pub fn try_acquire(&self) -> Option<SemaphorePermit<'_>>

Try to acquire a permit without waiting.

Returns None if the limit is reached.

pub fn available_permits(&self) -> usize

Get the number of available semaphore permits.

pub fn config(&self) -> &ConcurrencyConfig

Get the configuration.

pub fn rate_limiter(&self) -> Option<&RateLimiter>

Get the rate limiter (if any).

Trait Implementations

impl Clone for ConcurrencyController

fn clone(&self) -> ConcurrencyController

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ConcurrencyController

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ConcurrencyController

fn default() -> Self

Returns the “default value” for a type.