Trait BackendSession 

pub trait BackendSession<B: BackendProvider> {
    type PrefillInput;
    type DecodeInput;
    type Output;
    type Completion: Completion<Error = B::Error>;

    // Required methods
    fn capabilities(&self) -> SessionCapabilities;
    fn prefill(
        &mut self,
        backend: &B,
        input: Self::PrefillInput,
    ) -> Result<Submission<Self::Output, Self::Completion>, B::Error>;
    fn decode(
        &mut self,
        backend: &B,
        input: Self::DecodeInput,
    ) -> Result<Submission<Self::Output, Self::Completion>, B::Error>;
    fn observe_output(
        &self,
        backend: &B,
        output: &Self::Output,
    ) -> Result<ObservationSet, B::Error>;
}

Prefill/decode interface for an already selected backend session.

The session owns its prepared executable and cache. The contract intentionally models language-model submissions rather than primitive tensor operations. Input, output, cache and completion stay opaque.

Required Associated Types

type PrefillInput

Backend-owned prefill input.

type DecodeInput

Backend-owned decode input.

type Output

Backend-owned logits/output.

type Completion: Completion<Error = B::Error>

Exact completion type.

Required Methods

fn capabilities(&self) -> SessionCapabilities

Reports capabilities of this exact realized session.

fn prefill( &mut self, backend: &B, input: Self::PrefillInput, ) -> Result<Submission<Self::Output, Self::Completion>, B::Error>

Submits prompt prefill against this session.

fn decode( &mut self, backend: &B, input: Self::DecodeInput, ) -> Result<Submission<Self::Output, Self::Completion>, B::Error>

Submits one or more cached decode positions against this session.

fn observe_output( &self, backend: &B, output: &Self::Output, ) -> Result<ObservationSet, B::Error>

Materializes an already completed opaque output into portable records.

Calling this method is an explicit synchronization and host-transfer boundary. Ordinary inference does not invoke it.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors