Struct Orchestrator 

pub struct Orchestrator { /* private fields */ }

Orchestrator manages vLLM process lifecycle

Implementations

impl Orchestrator

pub fn new(configs: HashMap<String, ModelConfig>) -> Self

Create a new orchestrator with the given model configurations

pub fn with_command( configs: HashMap<String, ModelConfig>, vllm_command: String, ) -> Self

Create a new orchestrator with a custom command for spawning processes

This is useful for testing with mock-vllm

pub fn with_options( configs: HashMap<String, ModelConfig>, vllm_command: String, checkpoint_config: Option<CheckpointConfig>, ) -> Self

Create a new orchestrator with full options including checkpoint config

pub async fn process_state(&self, model: &str) -> Option<ProcessState>

Get the current state of a model’s process

pub fn registered_models(&self) -> Vec<String>

Get all registered model names

pub fn sleep_level_for(&self, model: &str) -> Option<u8>

Get the configured sleep level for a model

pub async fn ensure_running(&self, model: &str) -> Result<(), OrchestratorError>

Ensure a model’s process is running and ready

This will:

1. Start the process if not started
2. Wait for it to become healthy
3. Return once the model is ready to serve requests

pub async fn wake_model(&self, model: &str) -> Result<(), OrchestratorError>

Wake a model from sleep

pub async fn sleep_model( &self, model: &str, level: SleepLevel, ) -> Result<(), OrchestratorError>

Put a model to sleep

pub async fn force_sleep(&self, model: &str, level: SleepLevel)

Force a model to sleep, escalating to Stop if the initial level fails.

This is a guaranteed-cleanup method: it logs errors but never returns Err. Used to clean up partially-woken models that hold GPU memory.

pub async fn is_ready(&self, model: &str) -> bool

Check if a model is ready

Trait Implementations

impl Drop for Orchestrator

fn drop(&mut self)

Executes the destructor for this type.