Struct EngineClient 

pub struct EngineClient { /* private fields */ }

A connection to one engine group: every attention-DP rank’s engine.

Dropping the client ends every open stream with StreamError::ClientClosed.

Implementations

impl EngineClient

pub async fn connect(config: ClientConfig) -> Result<Self>

Bind the frontend sockets and wait until every engine has registered. See ClientListener to learn the bound address before waiting.

pub fn engines(&self) -> Vec<EngineInfo>

The engines, in index order, with the configuration each registered.

pub fn loads(&self) -> Vec<EngineLoad>

Each engine’s latest reported load, in index order.

pub fn in_flight(&self) -> Vec<usize>

This client’s unfinished requests per engine, in index order.

pub async fn generate( &self, request: TokenizedGenerateReqInput, ) -> Result<RequestStream>

Submit a request and return its output stream.

The request’s rid must be a single string, unique among requests in flight. Dropping the stream before its last chunk aborts the request on the engine.

pub async fn control( &self, request: ControlRequest, ) -> Result<Vec<ControlOutput>>

Run a control operation on every engine and return their replies, one per engine, in arrival order. Operations of one kind run one at a time; some (a wait pause, a weight load) answer only when done, so callers bound the wait themselves.

pub async fn abort(&self, rid: &str) -> Result<bool>

Ask the engine to abort a request. Its stream then ends with an "abort" chunk. Returns false if the request is not in flight.

Trait Implementations

impl Drop for EngineClient

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.