Crate llmleaf_client

Expand description

§llmleaf-client

Official async Rust SDK for the llmleaf LLM proxy. It speaks llmleaf’s OpenAI/OpenRouter-shaped JSON over HTTP — the wire is JSON, never protobuf-binary — and covers every endpoint in clients/SPEC.md:

chat completions, non-streaming (Client::chat) and streaming (Client::chat_stream, an impl Stream of ChatCompletionChunks that stops on the [DONE] sentinel);
embeddings (Client::embeddings, with base64 → float decoding);
model catalog (Client::list_models);
text-to-speech (Client::speech → (bytes, content_type)) and the voice catalog (Client::voices);
speech-to-text (Client::transcribe, multipart with the file part);
batches (Client::create_batch, Client::get_batch, Client::cancel_batch, Client::batch_results → an NDJSON stream of BatchResultLines).

§Codegen

build.rs compiles proto/llmleaf/v1/llmleaf.proto with prost-build (real codegen, run every build); the generated messages are exposed under pb. The public types (re-exported at the crate root, e.g. ChatRequest) are hand-written serde structs that produce the exact OpenAI JSON wire — prost compiles the proto while serde drives the wire.

§Quickstart

use llmleaf_client::{Client, ChatRequest, ChatMessage};

let client = Client::new("https://gateway.example.com", "sk-...")?;
let resp = client
    .chat(ChatRequest::new(
        "gpt-4o-mini",
        vec![ChatMessage::user("hi")],
    ))
    .await?;
println!("{}", resp.first_text().unwrap_or_default());

Modules§

pb: Prost-generated typed model, compiled from proto/llmleaf/v1/llmleaf.proto by build.rs. This is the “real codegen” proof: the proto genuinely compiles into the crate and the generated messages are available here as a typed mirror.

Structs§

Architecture: Model architecture metadata.
BatchCounts: Aggregate counts on a batch handle.
BatchCreateRequest: POST /v1/batches request body.
BatchError: A failed per-item batch response.
BatchHandle: A batch handle returned by create / retrieve / cancel.
BatchRequestItem: One item in a batch create request: a custom_id plus a ChatRequest body.
BatchResponse: A successful per-item batch response.
BatchResultLine: One line of the JSONL results stream.
ChatCompletionChunk: A streaming SSE frame (object:"chat.completion.chunk").
ChatMessage: A chat message (request or response).
ChatRequest: POST /v1/chat/completions request body.
ChatResponse: POST /v1/chat/completions non-streaming response (object:"chat.completion").
Choice: One completion choice.
ChunkChoice: One streaming choice.
Client: Async client for the llmleaf gateway.
ClientBuilder: Builder for Client. Obtain via Client::builder.
Delta: Incremental delta on a streaming choice.
Embedding: A single embedding vector. The transport always presents embedding as floats, even when the wire carried base64.
EmbeddingRequest: POST /v1/embeddings request body.
EmbeddingResponse: POST /v1/embeddings response (object:"list").
FunctionCall: A function the model called. arguments is a JSON-encoded string (OpenAI shape).
FunctionCallDelta: Incremental tool-call fragment on a streaming delta.
FunctionDef: A function the model MAY call. parameters is a raw JSON Schema value.
FunctionName: Pin a specific function: {"name":"..."}.
ImageUrl: {"url":"...","detail":"auto"}.
ListModelsResponse: GET /v1/models response.
ModelEndpoint: Admin-only fallback-chain entry (present only with a valid x-admin-token).
ModelEntry: One model in the catalog.
NamedToolChoice: Named tool choice: {"type":"function","function":{"name":"..."}}.
Pricing: Per-token pricing (decimal strings, USD).
PromptTokensDetails: Breakdown of Usage::prompt_tokens. Today only the cache-read (hit) share is surfaced — the count of prompt tokens served from the provider’s cache rather than processed fresh.
ReasoningDetail: One structured reasoning (“thinking”) block (OpenRouter reasoning_details[]). It expresses both open reasoning — visible text, optionally signed — and hidden reasoning — an encrypted/redacted blob the provider returns in place of the text. kind (wire type) is the discriminator:
ResponseFormat: response_format: {"type":"text"|"json_object"|"json_schema","json_schema":{...}}.
SpeechRequest: POST /v1/audio/speech request body.
ToolCall: A tool call emitted by the model.
ToolCallDelta: Streaming tool-call delta; fields arrive piecemeal.
ToolDef: A tool definition ({"type":"function","function":{...}}).
TopProvider: Top-provider capabilities for a model.
TranscriptionRequest: The accompanying form fields for POST /v1/audio/transcriptions. The audio bytes are the multipart file part and are passed separately to the SDK call.
TranscriptionResponse: Structured transcription result (for response_format json / verbose_json).
Usage: Token accounting echoed on every response. cost_usd is an llmleaf addition and is absent when the model has no known price.
Voice: A TTS voice.
VoicesResponse: GET /v1/audio/voices response.

Enums§

BatchStatus: Batch lifecycle status.
Content: Message content: a plain string when there is only text, else an array of parts. Untagged so it serialises as a bare string or a bare array, matching the wire.
ContentPart: A single content part of a multimodal message.
EmbeddingInput: input: the wire accepts a bare string or an array of strings. Untagged.
Error: Errors returned by the SDK.
FinishReason: Why the model stopped generating.
ModelType: type query filter for GET /v1/models.
Role: Message author role.
Stop: stop: a bare string for one element, else an array (the wire accepts either; we emit a string for a single element and an array otherwise). Untagged.
ToolChoice: tool_choice: a bare mode string ("auto"/"none"/"required") or a named object. Untagged so it serialises as a bare string or the object, matching the wire.
Transcription: Result of a transcription: a structured object (json/verbose_json) or a plain-text body (text/srt/vtt) — SPEC.md returns text directly for the latter.

Type Aliases§

Result: A convenience result alias.