Skip to main content

Engine

ferrox_models::engine

Trait Engine 

Source 
pub trait Engine: Sync {
    type State: Send;

    // Required methods
    fn new_state(&self) -> Self::State;
    fn vocab_size(&self) -> usize;
    fn forward_token_on_worker(
        &self,
        token_id: usize,
        pos: usize,
        state: &mut Self::State,
    ) -> Vec<f32>;

    // Provided method
    fn forward_token(
        &self,
        token_id: usize,
        pos: usize,
        state: &mut Self::State,
    ) -> Vec<f32> { ... }
}
Expand description

A decoder that can run one incremental forward step given a token id and position, updating its own per-layer state in place.

§What an implementor writes

Engine::forward_token_on_worker, never Engine::forward_token. The second is provided, and providing it again is the one way to lose the pool promotion silently; see the module docs.

§Why Sync and State: Send

Engine::forward_token hands &self and &mut Self::State to a rayon worker for the duration of the step, which is exactly what those two bounds say. They are stated on the trait rather than as a where clause on the method so that an engine which cannot satisfy them fails to compile at its impl, where the author can see why, instead of at every call site of a method it silently would not have.

Required Associated Types§

Source

type State: Send

Required Methods§

Source

fn new_state(&self) -> Self::State

Builds fresh (empty) per-layer state for a new request.

Source

fn vocab_size(&self) -> usize

Source

fn forward_token_on_worker( &self, token_id: usize, pos: usize, state: &mut Self::State, ) -> Vec<f32>

One decode step, already running on a rayon worker.

This is the body an engine writes. Callers want Engine::forward_token instead: same computation, with the pool entered once for the whole step.

Provided Methods§

Source

fn forward_token( &self, token_id: usize, pos: usize, state: &mut Self::State, ) -> Vec<f32>

One decode step for token_id at position pos.

Enters the CPU worker pool once for the whole call, so every parallel region the step opens takes rayon’s in-worker path. See the module docs for what that is worth, and ferrox_core::par::on_workers for the three cases it declines to promote (a backend whose thread-affine state par::carry does not carry, a pinned spin pool, and a caller already on a worker).

Do not override this. The body is one line, and the only reason it is a provided method rather than a free function is that a free function could not be spelled engine.forward_token(..) by the generation loops that already spell it that way.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§