llmkit-tower 0.1.0

Tower middleware (retry, rate limit, cost tracking, tracing) for llmkit-rs
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

//! Tower-style middleware for `llmkit-rs`.
//!
//! Each layer wraps an [`llmkit_core::LlmProvider`] and is itself a provider, so
//! they compose into a single `Arc<dyn LlmProvider>` without `Service<Request>`
//! generics or sprawling `where` clauses. Layers:
//!
//! - [`RetryLayer`] — exponential backoff over retryable errors
//! - [`RateLimitLayer`] — token-bucket throttling per provider
//! - [`CostTrackingLayer`] — per-request + cumulative cost, optional budget cap
//! - [`TracingLayer`] — structured spans with latency and token counts
//!
//! [`FallbackProvider`] chains providers primary → secondary on failure.

#![forbid(unsafe_code)]
#![deny(missing_docs)]

mod cost;
mod fallback;
mod layer;
mod rate_limit;
mod retry;
mod tracing;

pub use cost::{CostTracking, CostTrackingLayer, SessionCost};
pub use fallback::FallbackProvider;
pub use layer::LlmLayer;
pub use rate_limit::{RateLimit, RateLimitLayer};
pub use retry::{Retry, RetryLayer};
pub use tracing::{Tracing, TracingLayer};