Enum RequestMsg 

pub enum RequestMsg {
    Dispatch {
        deployment: String,
        batch: ExecuteBatch,
    },
    Chunk(Result<TokenChunk, InferenceError>),
    Cancel,
}

Variants

Dispatch

Kick off the request: routes via the coordinator and dispatches to the chosen engine.

Fields

deployment: String

batch: ExecuteBatch

Chunk(Result<TokenChunk, InferenceError>)

Forwarded chunk from the engine.

Cancel

Gateway gave up on the response (client disconnected). Cancel.