Struct InferenceHealthCheck 

pub struct InferenceHealthCheck {
    pub consecutive_success: usize,
    pub consecutive_failure: usize,
    /* private fields */
}

Inference health check for LLM/AI backends.

Implements Pingora’s HealthCheck trait to integrate with the load balancing infrastructure. Verifies both server availability and model availability.

Fields

consecutive_success: usize

Consecutive successes needed to mark healthy

consecutive_failure: usize

Consecutive failures needed to mark unhealthy

Implementations

impl InferenceHealthCheck

pub fn new( endpoint: String, expected_models: Vec<String>, timeout: Duration, ) -> Self

Create a new inference health check.

Arguments

• endpoint - The models endpoint path (e.g., “/v1/models”)
• expected_models - List of model IDs that must be available
• timeout - Connection and response timeout

Trait Implementations

impl HealthCheck for InferenceHealthCheck

fn check<'life0, 'life1, 'async_trait>( &'life0 self, backend: &'life1 Backend, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Check if the backend is healthy.

Returns Ok(()) if healthy, Err with message if not.

fn health_threshold(&self, success: bool) -> usize

Return the health threshold for flipping health status.

• success: true - returns consecutive_success (unhealthy -> healthy)
• success: false - returns consecutive_failure (healthy -> unhealthy)

fn health_status_change<'life0, 'life1, 'async_trait>( &'life0 self, _target: &'life1 Backend, _healthy: bool, ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: Sync + 'async_trait,

Called when the health changes for a Backend.

fn backend_summary(&self, target: &Backend) -> String

Called when a detailed Backend summary is needed.