pub struct ModelEngine { /* private fields */ }Expand description
The real engine: a model, its state and its tokenizer, on one thread.
ForwardState is Send but not Sync — it owns the expert slot pool,
the io_uring ring and the pinned compute pool — so it is moved onto the
thread that serves requests and never shared. That is not a limitation
being worked around: concurrency here is one, structurally, because a
second decode loop would halve the token rate of both and there is exactly
one KV cache.
Implementations§
Source§impl ModelEngine
impl ModelEngine
Sourcepub fn new(
model: Model,
state: ForwardState,
tokenizer: RvmpTokenizer,
config: EngineConfig,
) -> Self
pub fn new( model: Model, state: ForwardState, tokenizer: RvmpTokenizer, config: EngineConfig, ) -> Self
Build an engine over an already-loaded model and state.
The loading is the caller’s, deliberately: state carries the cache
budget, thread pinning and prefill dials, and those belong to the
command line that set them rather than to this crate.
Trait Implementations§
Source§impl Engine for ModelEngine
impl Engine for ModelEngine
Source§fn prepare(
&mut self,
request: &ChatCompletionRequest,
) -> Result<Plan, ServerError>
fn prepare( &mut self, request: &ChatCompletionRequest, ) -> Result<Plan, ServerError>
Everything that can be refused before the response is committed. Read more
Source§fn run(
&mut self,
plan: Plan,
sink: &mut dyn TokenSink,
) -> Result<Completion, ServerError>
fn run( &mut self, plan: Plan, sink: &mut dyn TokenSink, ) -> Result<Completion, ServerError>
Run the plan, reporting through
sink. Read moreAuto Trait Implementations§
impl !Freeze for ModelEngine
impl !RefUnwindSafe for ModelEngine
impl !Sync for ModelEngine
impl !UnwindSafe for ModelEngine
impl Send for ModelEngine
impl Unpin for ModelEngine
impl UnsafeUnpin for ModelEngine
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more