Skip to main content

Module provider

Module provider 

Source
Expand description

Unified LLM provider abstraction (ADR-010).

One trait covers both local Candle inference and cloud frontier LLMs. The host picks a backend at session construction time — the rest of the SDK sees only LlmProvider. This is the seam that lets mobile apps swap local ↔ frontier without touching their UI code.

Design notes:

  • All types are plain std (no async runtime dep in this crate).
  • chat_stream uses a callback so each binding surface wraps it in its own async/stream primitive (FRB → Dart Stream, uniffi → async callback, wasm-bindgen → ReadableStream).
  • CredentialRef is a runtime value from the host — never embedded.

Structs§

ChatMessage
One turn in the conversation.
ChatRequest
A chat completion request. model is a routing hint:
ChatResponse
A completed chat response.
ChatToken
A single streamed token fragment.
CredentialRef
Runtime API credential. The host resolves this from platform keystore (Android Keystore / iOS Keychain) before calling start_session. The SDK never logs or persists the value.

Enums§

ChatRole
Which role a message belongs to.

Traits§

LlmProvider
The unified backend trait (ADR-010). Implemented by: