Skip to main content

TextGenerationBackend

eredu_core::backend

Trait TextGenerationBackend 

Source 
pub trait TextGenerationBackend: BackendProvider {
    type Prompt;
    type Token: TokenOutput<Error = Self::Error>;
    type TextGenerationState;
    type TextCompletion: Completion<Error = Self::Error>;

    // Required methods
    fn start_text_generation(
        backend: &Self,
        config: TextGenerationConfig,
    ) -> Result<Self::TextGenerationState, Self::Error>;
    fn prepare_text_prompt(
        backend: &Self,
        prompt_token_ids: Vec<u32>,
    ) -> Result<Self::Prompt, Self::Error>;
    fn submit_text_prefill(
        runtime: &mut ModelRuntime<Self>,
        prompt: Self::Prompt,
        filter: &TokenFilter,
        state: &mut Self::TextGenerationState,
    ) -> Result<Submission<Self::Token, Self::TextCompletion>, Self::Error>;
    fn submit_text_decode(
        runtime: &mut ModelRuntime<Self>,
        token: Self::Token,
        filter: &TokenFilter,
        state: &mut Self::TextGenerationState,
    ) -> Result<Submission<Self::Token, Self::TextCompletion>, Self::Error>;
}
Expand description

High-level text-generation extension implemented once per backend.

The contract deliberately combines model execution and sampling. Core does not see logits or ask a backend to implement tensor primitives. The token, sampling state, cache state, and exact completion remain backend-owned.

Required Associated Types§

Source

type Prompt

Opaque prepared prompt, including any backend-owned multimodal values.

Source

type Token: TokenOutput<Error = Self::Error>

Backend-owned generated token handle.

Source

type TextGenerationState

Backend-owned sampler and randomness state for one sequence.

Source

type TextCompletion: Completion<Error = Self::Error>

Exact completion retaining model execution and token sampling.

Required Methods§

Source

fn start_text_generation( backend: &Self, config: TextGenerationConfig, ) -> Result<Self::TextGenerationState, Self::Error>

Creates backend sampling state for one sequence.

Source

fn prepare_text_prompt( backend: &Self, prompt_token_ids: Vec<u32>, ) -> Result<Self::Prompt, Self::Error>

Converts portable tokenizer ids into a backend-owned text prompt.

Source

fn submit_text_prefill( runtime: &mut ModelRuntime<Self>, prompt: Self::Prompt, filter: &TokenFilter, state: &mut Self::TextGenerationState, ) -> Result<Submission<Self::Token, Self::TextCompletion>, Self::Error>

Submits prompt prefill followed by sampling one token.

Source

fn submit_text_decode( runtime: &mut ModelRuntime<Self>, token: Self::Token, filter: &TokenFilter, state: &mut Self::TextGenerationState, ) -> Result<Submission<Self::Token, Self::TextCompletion>, Self::Error>

Submits cached decode from the preceding token and samples its successor.

Dyn Compatibility§

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§