Struct InferenceRateLimitManager 

pub struct InferenceRateLimitManager { /* private fields */ }

Manager for inference rate limiting, budgets, and cost tracking.

Each route with inference configuration gets its own TokenRateLimiter, TokenBudgetTracker, and CostCalculator based on the route’s configuration.

Implementations

impl InferenceRateLimitManager

pub fn new() -> Self

Create a new inference rate limit manager

pub fn register_route(&self, route_id: &str, config: &InferenceConfig)

Register a route with inference configuration.

Creates a TokenRateLimiter, TokenBudgetTracker, and CostCalculator as configured for the route.

pub fn has_route(&self, route_id: &str) -> bool

Check if a route has inference configuration registered.

pub fn has_budget(&self, route_id: &str) -> bool

Check if a route has budget tracking enabled.

pub fn has_cost_attribution(&self, route_id: &str) -> bool

Check if a route has cost attribution enabled.

pub fn check( &self, route_id: &str, key: &str, headers: &HeaderMap, body: &[u8], ) -> Option<InferenceCheckResult>

Check rate limit for a request.

Returns the rate limit result and the estimated token count.

pub fn check_budget( &self, route_id: &str, tenant: &str, estimated_tokens: u64, ) -> Option<BudgetCheckResult>

Check budget for a request.

Returns the budget check result, or None if no budget is configured.

pub fn record_budget( &self, route_id: &str, tenant: &str, actual_tokens: u64, ) -> Vec<BudgetAlert>

Record budget usage after a request completes.

Returns any budget alerts that were triggered.

pub fn budget_status( &self, route_id: &str, tenant: &str, ) -> Option<TenantBudgetStatus>

Get budget status for a tenant.

pub fn calculate_cost( &self, route_id: &str, model: &str, input_tokens: u64, output_tokens: u64, ) -> Option<CostResult>

Calculate cost for a request.

Returns the cost result, or None if cost attribution is not configured.

pub fn record_actual( &self, route_id: &str, key: &str, headers: &HeaderMap, body: &[u8], estimated_tokens: u64, ) -> Option<TokenEstimate>

Record actual token usage from response.

This adjusts the rate limiter based on actual vs estimated usage.

pub fn route_count(&self) -> usize

Get the number of registered routes

pub fn route_stats(&self, route_id: &str) -> Option<InferenceRouteStats>

Get stats for a route.

pub fn cleanup(&self)

Clean up idle rate limiters (called periodically)

Trait Implementations

impl Default for InferenceRateLimitManager

fn default() -> Self

Returns the “default value” for a type.