Skip to main content

Engine

Trait Engine 

Source
pub trait Engine {
    // Required methods
    fn model_id(&self) -> &str;
    fn created(&self) -> u64;
    fn prepare(
        &mut self,
        request: &ChatCompletionRequest,
    ) -> Result<Plan, ServerError>;
    fn run(
        &mut self,
        plan: Plan,
        sink: &mut dyn TokenSink,
    ) -> Result<Completion, ServerError>;
}
Expand description

The generator behind the HTTP surface.

Required Methods§

Source

fn model_id(&self) -> &str

The model id /v1/models advertises and every response echoes.

Source

fn created(&self) -> u64

Unix seconds to report as the model’s creation time.

Source

fn prepare( &mut self, request: &ChatCompletionRequest, ) -> Result<Plan, ServerError>

Everything that can be refused before the response is committed.

§Errors

Any ServerError whose status is a 4xx: this is where a bad request becomes an HTTP status rather than a broken stream.

Source

fn run( &mut self, plan: Plan, sink: &mut dyn TokenSink, ) -> Result<Completion, ServerError>

Run the plan, reporting through sink.

A sink failure is not an error here: it produces a Completion with Completion::aborted set, because “the client left” is an ordinary outcome and the engine must be ready for the next request either way.

§Errors

ServerError::Generate or ServerError::Forward when the runtime itself fails. The engine is left able to serve the next request.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§