Crate llmleaf_client

Expand description

§llmleaf-client

Official async Rust SDK for the llmleaf LLM proxy. It speaks llmleaf’s OpenAI/OpenRouter-shaped JSON over HTTP — the wire is JSON, never protobuf-binary — and covers every endpoint in clients/SPEC.md:

chat completions, non-streaming (Client::chat) and streaming (Client::chat_stream, an impl Stream of ChatCompletionChunks that stops on the [DONE] sentinel);
the OpenAI Responses dialect, non-streaming (Client::responses) and streaming (Client::responses_stream, an impl Stream of ResponsesStreamEvents that ends on the terminal response.completed/incomplete/failed event — there is no [DONE] sentinel);
embeddings (Client::embeddings, with base64 → float decoding);
rerank (Client::rerank, scoring documents against a query);
model catalog (Client::list_models);
text-to-speech (Client::speech → (bytes, content_type)) and the voice catalog (Client::voices);
speech-to-text (Client::transcribe, multipart with the file part);
batches (Client::create_batch, Client::get_batch, Client::cancel_batch, Client::batch_results → an NDJSON stream of BatchResultLines).

§Codegen

build.rs compiles proto/llmleaf/v1/llmleaf.proto with prost-build (real codegen, run every build); the generated messages are exposed under pb. The public types (re-exported at the crate root, e.g. ChatRequest) are hand-written serde structs that produce the exact OpenAI JSON wire — prost compiles the proto while serde drives the wire.

§Quickstart

use llmleaf_client::{Client, ChatRequest, ChatMessage};

let client = Client::new("https://gateway.example.com", "sk-...")?;
let resp = client
    .chat(ChatRequest::new(
        "gpt-4o-mini",
        vec![ChatMessage::user("hi")],
    ))
    .await?;
println!("{}", resp.first_text().unwrap_or_default());

Modules§

pb: Prost-generated typed model, compiled from proto/llmleaf/v1/llmleaf.proto by build.rs. This is the “real codegen” proof: the proto genuinely compiles into the crate and the generated messages are available here as a typed mirror.

Structs§

Architecture: Model architecture metadata.
BatchCounts: Aggregate counts on a batch handle.
BatchCreateRequest: POST /v1/batches request body.
BatchError: A failed per-item batch response.
BatchHandle: A batch handle returned by create / retrieve / cancel.
BatchRequestItem: One item in a batch create request: a custom_id plus a ChatRequest body.
BatchResponse: A successful per-item batch response.
BatchResultLine: One line of the JSONL results stream.
ChatCompletionChunk: A streaming SSE frame (object:"chat.completion.chunk").
ChatMessage: A chat message (request or response).
ChatRequest: POST /v1/chat/completions request body.
ChatResponse: POST /v1/chat/completions non-streaming response (object:"chat.completion").
Choice: One completion choice.
ChunkChoice: One streaming choice.
Client: Async client for the llmleaf gateway.
ClientBuilder: Builder for Client. Obtain via Client::builder.
Delta: Incremental delta on a streaming choice.
Embedding: A single embedding vector. The transport always presents embedding as floats, even when the wire carried base64.
EmbeddingRequest: POST /v1/embeddings request body.
EmbeddingResponse: POST /v1/embeddings response (object:"list").
FunctionCall: A function the model called. arguments is a JSON-encoded string (OpenAI shape).
FunctionCallDelta: Incremental tool-call fragment on a streaming delta.
FunctionDef: A function the model MAY call. parameters is a raw JSON Schema value.
FunctionName: Pin a specific function: {"name":"..."}.
ImageUrl: {"url":"...","detail":"auto"}.
ListModelsResponse: GET /v1/models response.
ModelEndpoint: Admin-only fallback-chain entry (present only with a valid x-admin-token).
ModelEntry: One model in the catalog.
NamedToolChoice: Named tool choice: {"type":"function","function":{"name":"..."}}.
Pricing: Per-token pricing (decimal strings, USD).
PromptTokensDetails: Breakdown of Usage::prompt_tokens. Today only the cache-read (hit) share is surfaced — the count of prompt tokens served from the provider’s cache rather than processed fresh.
ReasoningDetail: One structured reasoning (“thinking”) block (OpenRouter reasoning_details[]). It expresses both open reasoning — visible text, optionally signed — and hidden reasoning — an encrypted/redacted blob the provider returns in place of the text. kind (wire type) is the discriminator:
RerankRequest: POST /v1/rerank request body.
RerankResponse: POST /v1/rerank response (object:"list").
RerankResult: A single reranked document: its position in the request documents and its score.
ResponseFormat: response_format: {"type":"text"|"json_object"|"json_schema","json_schema":{...}}.
ResponseFunctionCallItem: A function call the model made ({"type":"function_call","call_id","name","arguments"}). call_id pairs it with its ResponseFunctionCallOutputItem; arguments is the raw JSON string exactly as emitted.
ResponseFunctionCallOutputItem: The caller’s answer to a function call, replayed on the next turn ({"type":"function_call_output","call_id":"...","output":"..."}).
ResponseMessageItem: A conversation message item. On the wire it is a role-keyed object with NO "type" field: {"role":"user","content":"..."}. id/status are output-only.
ResponseReasoningItem: A reasoning (“thinking”) item. encrypted_content is opaque and MUST be echoed back verbatim in the next request’s input to continue an encrypted reasoning turn (SPEC.md).
ResponseReasoningText: One reasoning text entry. Its wire "type" token is decided by the list it sits in on a ResponseReasoningItem: summary[] → "summary_text", content[] → "reasoning_text".
ResponsesError: The inline error body carried on a failed ResponsesResponse (status:"failed"); the proto’s ErrorBody. Distinct from an HTTP error envelope (which surfaces as crate::Error::Api) and from the streaming "error" event.
ResponsesIncompleteDetails: status:"incomplete" refinement: "max_output_tokens" | "content_filter".
ResponsesInputTokensDetails: Prompt-cache hit accounting inside ResponsesUsage.
ResponsesNamedToolChoice: The FLAT named tool choice: {"type":"function","name":"..."} (no nested function).
ResponsesOutputTokensDetails: Reasoning-token accounting inside ResponsesUsage.
ResponsesReasoning: reasoning: {"effort":"minimal"|"low"|"medium"|"high"|..., "summary":...}.
ResponsesRequest: POST /v1/responses request body.
ResponsesResponse: The response object (object:"response"), also the snapshot carried by the response.created / response.in_progress / terminal stream events. Wire fields this SDK does not model (tools, truncation, parallel_tool_calls, …) are ignored on decode.
ResponsesStreamEvent: One streaming SSE event. Unlike chat streaming there is NO data: [DONE] sentinel: the stream ends after the terminal response.completed / response.incomplete / response.failed event. This is a flat superset of every event’s fields — kind says which are meaningful; see the accessor helpers.
ResponsesToolDef: A tool the model MAY call — FLAT in this dialect (type/name/parameters at the top level, no nested function object). parameters is a raw JSON Schema value.
ResponsesUsage: Token accounting in the Responses dialect’s own names (input_tokens/output_tokens, not the chat dialect’s prompt_tokens/completion_tokens).
SpeechRequest: POST /v1/audio/speech request body.
ToolCall: A tool call emitted by the model.
ToolCallDelta: Streaming tool-call delta; fields arrive piecemeal.
ToolDef: A tool definition ({"type":"function","function":{...}}).
TopProvider: Top-provider capabilities for a model.
TranscriptionRequest: The accompanying form fields for POST /v1/audio/transcriptions. The audio bytes are the multipart file part and are passed separately to the SDK call.
TranscriptionResponse: Structured transcription result (for response_format json / verbose_json).
Usage: Token accounting echoed on every response. cost_usd is an llmleaf addition and is absent when the model has no known price.
Voice: A TTS voice.
VoicesResponse: GET /v1/audio/voices response.

Enums§

BatchStatus: Batch lifecycle status.
Content: Message content: a plain string when there is only text, else an array of parts. Untagged so it serialises as a bare string or a bare array, matching the wire.
ContentPart: A single content part of a multimodal message.
EmbeddingInput: input: the wire accepts a bare string or an array of strings. Untagged.
Error: Errors returned by the SDK.
FinishReason: Why the model stopped generating.
ModelType: type query filter for GET /v1/models.
RerankDocument: A rerank document. Usually a plain string; the wire also accepts a structured multimodal object ({ text?, image? }), captured here as raw JSON. Untagged, so it round-trips either shape (mirrors how EmbeddingInput stays JSON-flexible).
ResponseContent: A Responses message content: a bare string, or an array of parts. Untagged.
ResponseContentPart: A single content part of a Responses message. The wire "type" token matches the set variant; note input_image.image_url is a plain string (not the chat dialect’s {url} object) and output_text always carries an annotations array.
ResponseItem: One item of the request input array or the response output array. The wire discriminator is "type"; a message is the role-keyed object emitted with no "type" (and decoded from an absent or "message" type). ResponseItem::Other preserves any item type this SDK version does not model, verbatim.
ResponsesInput: input: a bare string (one user message) or an array of items. Untagged so it serialises as a bare string or a bare array, matching the wire.
ResponsesToolChoice: tool_choice: a bare mode string ("auto"/"none"/"required") or the flat named object. Untagged so it serialises as a bare string or the object, matching the wire.
Role: Message author role.
Stop: stop: a bare string for one element, else an array (the wire accepts either; we emit a string for a single element and an array otherwise). Untagged.
ToolChoice: tool_choice: a bare mode string ("auto"/"none"/"required") or a named object. Untagged so it serialises as a bare string or the object, matching the wire.
Transcription: Result of a transcription: a structured object (json/verbose_json) or a plain-text body (text/srt/vtt) — SPEC.md returns text directly for the latter.

Type Aliases§

Result: A convenience result alias.