Module rate_limiter 

Lock-free rate limiter for high-performance request throttling.

This module provides a hierarchical rate limiter that uses atomic operations for zero-contention request rate limiting. Unlike traditional mutex-based rate limiters, this implementation uses compare-and-swap primitives to achieve 1M+ checks/second with minimal overhead.

Features

• Lock-free: Uses atomic operations (compare-and-swap) for zero lock contention
• Hierarchical: Per-provider, per-model, per-user rate limits
• Adaptive: Automatically adjusts based on usage patterns
• High performance: 1M+ rate checks per second
• Memory efficient: Minimal memory overhead per limiter

Algorithm

Uses a token bucket algorithm with atomic operations:

• Each limiter maintains current tokens and last refill timestamp
• Check operation: CAS (compare-and-swap) atomic to decrement tokens
• Refill happens lazily when tokens are checked

Structs

RateLimiter

Lock-free token bucket rate limiter

TokenBucketConfig

Token bucket rate limit configuration