Struct GenerateRequest 

pub struct GenerateRequest { /* private fields */ }

Generates text with the native endpoint. Build with Client::generate.

Implementations

impl GenerateRequest

pub fn gpu(self, gpu: impl Into<String>) -> Self

Route to a machine profile, optionally pool-qualified as "pool/profile".

pub fn wait_for_capacity(self, wait: bool) -> Self

Whether to wait out provisioning and model loading, or fail fast.

pub fn provision_timeout(self, timeout: Duration) -> Self

Total wall-clock budget for this call, retries included.

pub fn max_oom_retries(self, retries: u32) -> Self

Cap on RESOURCE_EXHAUSTED retries. Zero fails fast.

pub fn temperature(self, value: f64) -> Self

Sampling temperature.

pub fn top_p(self, value: f64) -> Self

Nucleus sampling cutoff.

pub fn frequency_penalty(self, value: f64) -> Self

Penalty applied per prior occurrence of a token.

pub fn presence_penalty(self, value: f64) -> Self

Penalty applied once a token has appeared at all.

pub fn grammar(self, value: Grammar) -> Self

Constrain the output’s shape.

pub fn seed(self, value: i64) -> Self

Seed the sampler for reproducible output.

pub fn logit_bias(self, value: Value) -> Self

Per-token logit adjustments, as a {token_id: bias} object.

pub fn options(self, value: Value) -> Self

Adapter-specific runtime options.

pub fn routing_key(self, value: impl Into<String>) -> Self

Pin related requests to the same worker, to reuse its prefix cache.

pub fn prompt_cache_key(self, value: impl Into<String>) -> Self

Explicit prefix-cache key.

pub fn safety_identifier(self, value: impl Into<String>) -> Self

End-user identifier forwarded to safety tooling.

pub fn lora_adapter(self, value: impl Into<String>) -> Self

LoRA adapter to apply.

pub fn stop(self, stop: impl IntoIterator<Item = impl Into<String>>) -> Self

Sequences that end the generation.

pub fn images(self, images: impl IntoIterator<Item = ImageInput>) -> Self

Attach images to a multimodal prompt.

pub fn extra_body(self, extra: Value) -> Self

Merge extra fields into the request body, overriding anything the builder set.

pub async fn send(self) -> Result<GenerateResult>

Send the request and wait for the whole generation.

pub async fn estimate(self) -> Result<CostEstimate>

Estimate what this request would cost.

impl GenerateRequest

pub fn stream(self) -> Result<ChunkStream<GenerateChunk>>

Stream the generation token by token.