Expand description
Optimized LLM client with connection pooling and request batching
Structsยง
- Batched
Request - A request that can be batched with similar requests
- Cached
Response - Cached response with TTL
- Client
Metrics - Client performance metrics
- Connection
Pool - Connection pool for HTTP clients
- OptimizedLLM
Client - Optimized LLM client with advanced performance features
- Optimized
Request - Simplified request structure for optimization
- Optimized
Response - Simplified response structure
- Rate
Limiter - Rate limiter for API requests
- Request
Batcher - Request batching manager for similar requests
- Token
Bucket - Token bucket for rate limiting