Enum BatchRequest 

pub enum BatchRequest {
    Generate {
        prompt: String,
        max_tokens: usize,
        config: SamplerConfig,
        cache_prompt: bool,
        lora_selection: LoraSelection,
        reply: Sender<Result<(String, UsageStats), String>>,
    },
    GenerateStream {
        prompt: String,
        max_tokens: usize,
        config: SamplerConfig,
        cache_prompt: bool,
        lora_selection: LoraSelection,
        callback: StreamCallback,
        reply: Sender<Result<UsageStats, String>>,
    },
    Embed {
        text: String,
        reply: Sender<Result<Vec<f32>, String>>,
    },
}

A single inference request dispatched to the worker task.

Variants

Generate

Non-streaming generation: prompt → full response string.

Fields

prompt: String

The formatted prompt to generate from.

max_tokens: usize

Maximum number of tokens to generate.

config: SamplerConfig

Per-request sampler configuration.

cache_prompt: bool

Whether to look up and store the prompt’s KV state in the prefix cache. When true (default), the worker checks for a matching cached prefix and skips the redundant prefill if found.

lora_selection: LoraSelection

LoRA adapters to apply for this request. Empty means no LoRA.

reply: Sender<Result<(String, UsageStats), String>>

Channel to send the result back to the caller.

GenerateStream

Streaming generation: invokes callback for every decoded token.

Fields

prompt: String

The formatted prompt to generate from.

max_tokens: usize

Maximum number of tokens to generate.

config: SamplerConfig

Per-request sampler configuration.

cache_prompt: bool

Whether to look up and store the prompt’s KV state in the prefix cache.

lora_selection: LoraSelection

LoRA adapters to apply for this request. Empty means no LoRA.

callback: StreamCallback

Called with each token text inside the blocking worker thread.

reply: Sender<Result<UsageStats, String>>

Channel that receives Ok(UsageStats) once generation is complete, or Err(message) on failure.

Embed

Embedding computation: text → L2-normalised vector.

Fields

text: String

The text to embed.

reply: Sender<Result<Vec<f32>, String>>

Channel to return the embedding vector (or an error message).

Trait Implementations

impl Debug for BatchRequest

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.