pub trait Engine {
// Required methods
fn model_id(&self) -> &str;
fn created(&self) -> u64;
fn prepare(
&mut self,
request: &ChatCompletionRequest,
) -> Result<Plan, ServerError>;
fn run(
&mut self,
plan: Plan,
sink: &mut dyn TokenSink,
) -> Result<Completion, ServerError>;
}Expand description
The generator behind the HTTP surface.
Required Methods§
Sourcefn prepare(
&mut self,
request: &ChatCompletionRequest,
) -> Result<Plan, ServerError>
fn prepare( &mut self, request: &ChatCompletionRequest, ) -> Result<Plan, ServerError>
Everything that can be refused before the response is committed.
§Errors
Any ServerError whose status is a 4xx: this is where a bad request
becomes an HTTP status rather than a broken stream.
Sourcefn run(
&mut self,
plan: Plan,
sink: &mut dyn TokenSink,
) -> Result<Completion, ServerError>
fn run( &mut self, plan: Plan, sink: &mut dyn TokenSink, ) -> Result<Completion, ServerError>
Run the plan, reporting through sink.
A sink failure is not an error here: it produces a Completion with
Completion::aborted set, because “the client left” is an ordinary
outcome and the engine must be ready for the next request either way.
§Errors
ServerError::Generate or ServerError::Forward when the runtime
itself fails. The engine is left able to serve the next request.
Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".