Struct ChatRequest 

pub struct ChatRequest { /* private fields */ }

Generates a chat completion. Build with Client::chat.

Implementations

impl ChatRequest

pub fn gpu(self, gpu: impl Into<String>) -> Self

Route to a machine profile, optionally pool-qualified as "pool/profile".

pub fn wait_for_capacity(self, wait: bool) -> Self

Whether to wait out provisioning and model loading, or fail fast.

pub fn provision_timeout(self, timeout: Duration) -> Self

Total wall-clock budget for this call, retries included.

pub fn max_oom_retries(self, retries: u32) -> Self

Cap on RESOURCE_EXHAUSTED retries. Zero fails fast.

pub fn max_completion_tokens(self, value: u32) -> Self

Cap on generated tokens. Prefer this over max_tokens.

pub fn max_tokens(self, value: u32) -> Self

Legacy cap on generated tokens.

pub fn temperature(self, value: f64) -> Self

Sampling temperature.

pub fn top_p(self, value: f64) -> Self

Nucleus sampling cutoff.

pub fn top_k(self, value: u32) -> Self

Sample only from the k most likely tokens.

pub fn repetition_penalty(self, value: f64) -> Self

Penalty applied to repeated tokens.

pub fn tools(self, value: Vec<Value>) -> Self

Tool definitions the model may call.

pub fn tool_choice(self, value: Value) -> Self

Which tool the model must call, if any.

pub fn parallel_tool_calls(self, value: bool) -> Self

Whether several tool calls may be issued at once.

pub fn response_format(self, value: Value) -> Self

Structured output format.

pub fn frequency_penalty(self, value: f64) -> Self

Penalty applied per prior occurrence of a token.

pub fn presence_penalty(self, value: f64) -> Self

Penalty applied once a token has appeared at all.

pub fn n(self, value: u32) -> Self

Number of candidates to return.

pub fn best_of(self, value: u32) -> Self

Number of candidates to generate before picking. Incompatible with streaming.

pub fn logprobs(self, value: bool) -> Self

Return per-token log probabilities.

pub fn top_logprobs(self, value: u32) -> Self

How many alternatives to report per token, from 0 to 20.

pub fn logit_bias(self, value: Value) -> Self

Per-token logit adjustments, each from -100 to 100.

pub fn seed(self, value: i64) -> Self

Seed the sampler for reproducible output.

pub fn stream_options(self, value: Value) -> Self

Stream tuning, such as {"include_usage": true}.

pub fn user(self, value: impl Into<String>) -> Self

End-user identifier.

pub fn safety_identifier(self, value: impl Into<String>) -> Self

End-user identifier forwarded to safety tooling.

pub fn lora_adapter(self, value: impl Into<String>) -> Self

LoRA adapter to apply.

pub fn stop(self, stop: impl IntoIterator<Item = impl Into<String>>) -> Self

Sequences that end the generation.

pub fn extra_body(self, extra: Value) -> Self

Merge extra fields into the request body, overriding anything the builder set.

The gateway rejects unknown fields, so this is for options newer than the SDK.

pub async fn send(self) -> Result<ChatCompletion>

Send the request and wait for the whole completion.

pub async fn estimate(self) -> Result<CostEstimate>

Estimate what this request would cost.

impl ChatRequest

pub fn stream(self) -> Result<ChunkStream<ChatCompletionChunk>>

Stream the completion token by token.