Struct ServicePool 

pub struct ServicePool { /* private fields */ }

A pool of Tower service stacks, one per configured model name.

Each model name maps to a type-erased BoxCloneService with the full middleware stack applied (cache, health check, cooldown, rate limit, cost tracking, budget, tracing).

Implementations

impl ServicePool

pub fn from_config(config: &ProxyConfig) -> Result<Self, String>

Build a pool from the proxy configuration.

Groups config.models by name and creates a Tower service stack for each unique model name. When multiple deployments share a name, the first entry is used (round-robin load balancing is planned for v2).

Errors

Returns an error string if a DefaultClient cannot be constructed for any model entry.

pub fn get_service( &self, model: &str, ) -> Result<BoxCloneService<LlmRequest, LlmResponse, LiterLlmError>, ProxyError>

Clone and return a Tower service stack for the given model name.

Errors

Returns ProxyError::not_found if no model with that name exists.

pub fn get_client(&self, model: &str) -> Result<Arc<DefaultClient>, ProxyError>

Return a reference to the raw DefaultClient for the given model.

Useful for File, Batch, and Response API operations that bypass the Tower middleware stack.

Errors

Returns ProxyError::not_found if no model with that name exists.

pub fn first_client(&self) -> Result<Arc<DefaultClient>, ProxyError>

Return the first available raw client.

Used by File, Batch, and Response API endpoints that do not carry a model field in the request body.

pub fn model_names(&self) -> Vec<&str>

Return the names of all available models.

pub fn has_any_service(&self) -> bool

Return true if the pool contains at least one service.