Struct InMemoryRings 

pub struct InMemoryRings { /* private fields */ }

Implementations

impl InMemoryRings

pub fn new() -> Self

pub fn set_yield(&mut self, on: bool)

What yield_requested answers, for tests of an engine’s early stop.

pub fn set_latency(&mut self, on: bool)

What latency_wanted answers, for tests.

pub fn stage_prompt(&mut self, tokens: &[u32]) -> TokenRef

Trait Implementations

impl Debug for InMemoryRings

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for InMemoryRings

fn default() -> Self

Returns the “default value” for a type.

impl Rings for InMemoryRings

fn read_tokens(&self, r: &TokenRef) -> Result<Vec<u32>, RingError>

fn write_tokens(&mut self, tokens: &[u32]) -> Result<TokenRef, RingError>

fn write_logits_row(&mut self, row: &[f32]) -> RingRef

fn read_logits_row(&self, r: &RingRef) -> Result<Vec<f32>, RingError>

fn yield_requested(&self) -> bool

The daemon wants the running tick to end at its next step boundary (a request is waiting to be admitted, or a lane was cancelled).

fn latency_wanted(&self) -> bool

Latency-sensitive requests have arrived lately: run prefill in short steps, so a yield comes soon after it is asked for.

impl StageRings for InMemoryRings

fn stage_prompt(&mut self, tokens: &[u32]) -> TokenRef