Crate throttle_net 

throttle-net

Outbound throttling and resilience. Where rate-net protects your service from being overwhelmed (inbound), throttle-net protects your service from overwhelming the downstreams it calls — and from being banned by them. The defining operation is therefore to wait, not to reject: you pace your own outbound work rather than dropping someone else’s request.

throttle-net does not reimplement token-bucket accounting. It consumes better-bucket for that and reads time from clock-lib, then builds the waiting, cost-aware, composable surface on top. It is the outbound companion to rate-net.

Status

Stable (v1.0). The public API is frozen until 2.0. The limiter and resilience surface: the Limiter trait, the Throttle token bucket and the exact SlidingWindowLog, each with a waiting cost-aware acquire; the composites — Hybrid (must pass all), MultiLimiter (multi-dimensional budgets), PerKey (independent per-key state, bounded memory), and Layered (global / per-key / per-endpoint scopes); standalone Retry/Backoff with jittered backoff and Retry-After parsing; the resilience layer — a CircuitBreaker that wraps any limiter and fails fast (circuit-breaker feature), and a deadline-aware, priority Queue; adaptive concurrency — an AdaptiveLimiter that discovers the right in-flight limit from outcome feedback (adaptive feature); provider integration — response-header parsers with limiter sync (provider-headers feature) and LLM tier presets (provider-llm feature); and observability — metrics and tracing events, feature-gated and zero-cost when off (metrics, tracing features).

The waiting surface runs on either tokio or smol — the async code is runtime-agnostic, and you choose the timer backend by feature. With std off, the pure algorithm core (Backoff, Jitter, Decision) compiles no_std. Correctness is held by property tests for every limiter invariant, a loom model check of the lock-free slot accounting, and fuzzed parsers.

use throttle_net::Throttle;

// 100 requests per second, bursting up to 100.
let throttle = Throttle::per_second(100);

// Pace an outbound call: returns as soon as a token is free.
throttle.acquire().await?;
// ... call the downstream ...

When you would rather not wait, ask without blocking:

use throttle_net::Throttle;

let throttle = Throttle::per_second(100);
if throttle.try_acquire() {
    // a token was free — send now
}

Design goals

• Wait by default. The Tier-1 acquire paces the caller; try_acquire is there when you need the non-blocking answer.
• Cost-aware. Not every request weighs one unit. acquire_with_cost(n) takes n tokens at once — the basis for the multi-dimensional LLM budgets that arrive with the rest of v0.2.
• Lock-free accounting. Each acquire is a single atomic compare-and-swap in better-bucket; no lock sits on the path.
• Runtime-free core, lazy refill. Tokens accrue from a monotonic clock on access; there is no background timer thread, and the synchronous core has no async-runtime dependency.
• Composable. Every limiter is one Limiter; composites combine them without the call site changing.

Feature flags

Feature	Default	Description
std	yes	Standard library — the limiter surface. With it off the crate is no_std and exposes the pure algorithm core (Backoff, Jitter, Decision) plus VERSION.
tokio	yes	tokio timer backend for the waiting acquire surface. Implies std.
smol	no	smol timer backend, as an alternative to tokio. (async-std is unsupported — it is discontinued.)

The circuit-breaker, adaptive, provider-headers / provider-llm, and metrics / tracing features are documented in docs/API.md, which carries the full feature matrix.

Modules

presetsprovider-llm

Ready-made MultiLimiter configurations for common LLM provider tiers.

providerprovider-headers

Parse rate-limit response headers and reconcile a limiter with the server.

Structs

AdaptiveLimiteradaptive

A concurrency limiter whose in-flight limit adapts to observed outcomes.

AdaptiveLimiterBuilderadaptive

Builder for an AdaptiveLimiter.

AdaptivePermitadaptive

A reserved concurrency slot. Settle it with success or failure after the request completes; dropping it unsettled records a failure and frees the slot.

Aimdadaptive

Additive-increase, multiplicative-decrease.

Backoff

A backoff policy: a base curve plus a jitter mode and a delay ceiling.

BackoffIter

A live delay sequence produced by a Backoff.

CircuitBreakercircuit-breaker

A circuit breaker wrapping a limiter L, timed by clock C.

CircuitBreakerBuildercircuit-breaker

Builder for a CircuitBreaker.

Evictionstd

How a PerKey limiter bounds the memory its per-key state can occupy.

Hybridstd

Several limiters combined so a request must satisfy all of them.

HybridBuilderstd

Builder for a Hybrid limiter.

Layeredstd

Several scopes of limiting stacked so a request must clear every one.

LayeredBuilderstd

Builder for a Layered limiter.

ManualClockstd

A clock under your control, for deterministic testing.

MultiLimiterstd

A limiter with several named dimensions, each metered independently.

MultiLimiterBuilderstd

Builder for a MultiLimiter.

PerKeystd

A throttle that keeps independent state per key.

Permitcircuit-breaker

A reserved permission to make one protected call.

Queueruntime

A bounded, deadline-aware, priority queue fronting a limiter L, keyed by K for fairness and timed by clock C.

QueueBuilderruntime

Builder for a Queue.

Retrystd

A retry policy: a Backoff, an attempt ceiling, and whether to honor a server’s Retry-After.

SlidingWindowLogstd

An exact sliding-window-log rate limiter: at most limit units in any trailing window of window.

SystemClockstd

A clock backed by the operating system.

Throttlestd

A single outbound throttle backed by a token bucket.

Vegasadaptive

Latency-based adaptation, after TCP Vegas.

Enums

BreakerStatecircuit-breaker

The breaker’s current state, as a snapshot.

Decision

What happened when a limiter was asked for tokens without waiting.

Jitter

How retry delays are randomized to avoid synchronized retries.

Outcomeadaptive

The observed result of one completed request, fed back to the strategy.

Overflowruntime

What a full queue does with a new request.

RetryActionstd

What to do with an error a retried operation returned.

ThrottleErrorstd

An acquisition that cannot complete.

Tripcircuit-breaker

The condition under which a closed breaker trips open.

Constants

DEFAULT_MAX_KEYSstd

The default key-capacity cap, applied unless overridden.

VERSION

The version of this crate, from Cargo.toml.

Traits

AdaptiveStrategyadaptive

How an AdaptiveLimiter moves its concurrency limit in response to an Outcome.

Clockstd

A source of time.

Limiterstd

The contract a limiter satisfies: take a cost and report the Decision.

Functions

parse_retry_afterstd

Parses a Retry-After header value into a delay from now.

parse_retry_after_atstd

Parses a Retry-After value relative to an explicit current time, given as Unix seconds. The date form needs a reference point; this lets tests and clock-injecting callers supply one.

retry_if_retryablestd

Classifies an error_forge::ForgeError by its own retryability: retry when is_retryable is true, otherwise give up. A convenient classify argument for Retry::run.