MiniLLMLib-RS

A minimalist, async-first Rust library for LLM interactions with streaming support.

Documentation

Guide (tutorials, patterns, custom providers): https://weavemindai.github.io/MiniLLMLibRS/
API reference (every type and method): https://docs.rs/minillmlib

Features

Async-first: Built on Tokio for high-performance async operations
Streaming Support: First-class SSE streaming for real-time responses
Conversation Trees: ChatNode provides tree-based conversation structure with branching
Tree Manipulation: detach(), merge(), tree iterators (depth-first, breadth-first, leaves)
Template Substitution: Format kwargs with {placeholders} in messages
Thread Serialization: Save/load conversation threads to/from JSON files
Cost Tracking: OpenRouter usage accounting with callbacks
Cost Estimation: what a call will cost before it is sent, so you can decide whether to allow it (opt-in estimate feature)
Tool Calling: Normalized ToolDefinition/ToolChoice/ToolCall types; each provider emits its own wire (OpenAI tools, Anthropic tool_use), streaming included
Multimodal: Support for images, audio and video in messages
JSON Repair: Robust handling of malformed JSON from LLM outputs
OpenRouter Compatible: Works with OpenRouter, OpenAI, and any OpenAI-compatible API
Retry with Backoff: Built-in exponential backoff and retry logic
Provider Routing: OpenRouter provider settings (sort, ignore, data collection)

Installation

Add to your Cargo.toml:

[dependencies]
minillmlib = "0.5"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

Optional features

Feature	Default	What it adds
`estimate`	off	Cost estimation before a call is sent. Embeds a 1.6 MB token vocabulary, which a plain completion has no use for.
`live`	off	Integration tests that make real, billed API calls.

Quick Start

use minillmlib::{ChatNode, GeneratorInfo};

#[tokio::main]
async fn main() -> minillmlib::Result<()> {
    // Load .env and configure logging
    minillmlib::init();

    // Create a generator for OpenRouter
    let generator = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");

    // Start a conversation
    let root = ChatNode::root("You are a helpful assistant.");
    let response = root.chat("Hello!", &generator).await?;

    println!("Assistant: {}", response.text().unwrap_or_default());
    Ok(())
}

Environment Variables

Set your API key in a .env file or environment:

OPENROUTER_API_KEY=sk-or-v1-your-key-here
# Or for direct OpenAI:
OPENAI_API_KEY=sk-your-key-here

Usage Examples

Basic Completion

use minillmlib::{ChatNode, GeneratorInfo, CompletionParameters, NodeCompletionParameters};

let generator = GeneratorInfo::openrouter("anthropic/claude-3.5-sonnet");
let root = ChatNode::root("You are helpful.");
let user = root.add_user("What is 2+2?");

// With custom parameters
let params = NodeCompletionParameters::new()
    .with_params(
        CompletionParameters::new()
            .with_temperature(0.0)
            .with_max_tokens(100)
    );

let response = user.complete(&generator, Some(&params)).await?;
println!("{}", response.text().unwrap());

Streaming

let root = ChatNode::root("You are helpful.");
let user = root.add_user("Tell me a story.");

let mut stream = user.complete_streaming(&generator, None).await?;

while let Some(chunk) = stream.next_chunk().await {
    print!("{}", chunk?.delta);
}

Multi-turn Conversation

let root = ChatNode::root("You are helpful.");

// First turn
let response1 = root.chat("My name is Alice.", &generator).await?;

// Second turn - context is preserved
let response2 = response1.chat("What's my name?", &generator).await?;
// Response will mention "Alice"

Image Input

use minillmlib::{ChatNode, GeneratorInfo, ImageData, MessageContent};

let generator = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");
let image = ImageData::from_file("./image.jpg")?;

let content = MessageContent::with_images("Describe this image.", &[image]);
let root = ChatNode::root("You are helpful.");
let user = root.add_user(content);

let response = user.complete(&generator, None).await?;

Audio Input

use minillmlib::{AudioData, MessageContent};

let audio = AudioData::from_file("./audio.mp3")?;
let content = MessageContent::with_audio("Transcribe this audio.", &[audio]);

Tool / Function Calling

Tools are normalized: define them once, and each provider emits its own wire shape (OpenAI-wire tools/tool_calls, Anthropic tool_use/tool_result). See the guide's Tool Calling chapter for the full loop.

use minillmlib::{CompletionParameters, NodeCompletionParameters, ToolChoice, ToolDefinition};

let params = NodeCompletionParameters::new().with_params(
    CompletionParameters::new()
        .with_tool(ToolDefinition::new(
            "get_weather",
            "Get the current weather for a city",
            serde_json::json!({
                "type": "object",
                "properties": { "city": { "type": "string" } },
                "required": ["city"],
            }),
        ))
        .with_tool_choice(ToolChoice::Auto),
);

let node = user.complete(&generator, Some(&params)).await?;
if let Some(calls) = node.tool_calls() {
    let mut current = node.clone();
    for call in &calls {
        let args = call.arguments_json()?;                 // typed arguments
        let result = run_my_tool(&call.name, &args);       // your code
        current = current.add_tool_result(&call.id, result);
    }
    let answer = current.complete(&generator, Some(&params)).await?;
}

Streamed tool arguments can be consumed DECODED, live, per field: ArgumentStream gives every argument a FieldHandle where the consumer picks wait().await (complete parsed value) or delta().await (the field's text as the model generates it, escapes undone), with an opt-in lenient mode for sloppy models. See the guide's Tool Calling chapter.

For a complete multi-turn agent loop (streaming prose live, one streaming tool consuming its decoded payload as the model generates it, one buffered tool), run cargo run --example agent_loop (examples/agent_loop.rs).

JSON Response with Repair

let params = NodeCompletionParameters::new()
    .with_parse_json(true)           // Enable JSON repair
    .with_crash_on_refusal(true)     // Retry if no valid JSON
    .with_retry(3);                  // Number of retries

let response = user.complete(&generator, Some(&params)).await?;
// response.text() will contain valid, repaired JSON

Retry with Exponential Backoff

let params = NodeCompletionParameters::new()
    .with_retry(5)
    .with_exp_back_off(true)
    .with_back_off_time(1.0)    // Start with 1 second
    .with_max_back_off(30.0)    // Max 30 seconds
    .with_crash_on_empty(true); // Retry on empty responses

Force Prepend (Constrained Generation)

// Force the model to start its response with specific text
let params = NodeCompletionParameters::new()
    .with_force_prepend("Score: ");

// Response will start with "Score: " followed by the model's completion

OpenRouter Provider Settings

OpenRouter routing is provider-specific, so it's attached via with_openrouter_routing (which carries it under the request's provider key); non-OpenRouter providers simply ignore it.

use minillmlib::{CompletionParameters, ProviderSettings};

let routing = ProviderSettings::new()
    .sort_by_throughput()                              // or .sort_by_price()
    .deny_data_collection()
    .with_ignore(vec!["SambaNova".to_string()]);       // Exclude providers

let params = CompletionParameters::new()
    .with_openrouter_routing(routing);

Prompt Caching (provider-agnostic)

Mark what to cache on the tree; the provider decides the wire (Anthropic and OpenRouter-fronted Claude models emit cache_control breakpoints, OpenAI auto-caches). Switch the provider and the same code works.

let root = ChatNode::root(big_system_prompt);
root.cache_breakpoint();                 // cache just the system prompt
// ...or NodeCompletionParameters::new().with_cache(true) to cache the whole prefix

// Warm the cache before an agent run (cheap to call repeatedly):
let warm_cost = some_node.ensure_cached(&generator, None).await?;

// Clear marks:
root.clear_cache_breakpoint();           // one node
root.clear_all_cache_breakpoints();      // whole tree

Cache tokens are priced with distinct read/write rates (cache reads are ~0.1× input; cache writes a ~1.25× premium):

let price = TokenPrice::new(1.0, 5.0)        // $/Mtok input, output
    .with_cache_rates(0.1, 1.25);            // $/Mtok cache-read, cache-write

Custom/Extra Parameters

// Pass arbitrary parameters to the API
let params = CompletionParameters::new()
    .with_extra("custom_param", serde_json::json!(42))
    .with_extra("another", serde_json::json!({"nested": "value"}));

Pretty Print Conversations

use minillmlib::{pretty_messages, format_conversation, PrettyPrintConfig};

let root = ChatNode::root("You are helpful.");
let user = root.add_user("Hello");
let assistant = user.add_assistant("Hi there!");

// Default formatting
let pretty = format_conversation(&assistant);
// Output: "SYSTEM: You are helpful.\n\nUSER: Hello\n\nASSISTANT: Hi there!"

// Custom formatting
let config = PrettyPrintConfig::new("[SYS] ", "\n[USR] ", "\n[AST] ");
let pretty = pretty_messages(&assistant, Some(&config));

Template Substitution (Format Kwargs)

use minillmlib::ChatNode;

// Create a reusable prompt template
let root = ChatNode::root("You are {bot_name}, a {style} assistant.");
root.set_format_kwarg("bot_name", "Claude");
root.set_format_kwarg("style", "helpful");

let user = root.add_user("Hi {bot_name}!");

// Get formatted messages with placeholders replaced
let formatted = user.formatted_thread();
// Messages now contain "You are Claude, a helpful assistant." etc.

Save and Load Conversation Threads

use minillmlib::ChatNode;

// Build a conversation
let root = ChatNode::root("You are helpful.");
root.set_format_kwarg("name", "Alice");
let user = root.add_user("Hello {name}!");
let assistant = user.add_assistant("Hi there!");

// Save to JSON file
assistant.save_thread("conversation.json")?;

// Load from JSON file (returns root and leaf)
let (loaded_root, loaded_leaf) = ChatNode::from_thread_file("conversation.json")?;

// Or load from JSON string
let json = r#"{"prompts": [{"role": "system", "content": "Hello"}], "required_kwargs": {}}"#;
let (root, leaf) = ChatNode::from_thread_json(json)?;

Tree Manipulation

use minillmlib::ChatNode;

// Navigate to root from any node
let root = some_deep_node.get_root();

// Detach a subtree
let subtree = node.detach();  // node is now a new root

// Merge trees
let merged = tree1_leaf.merge(&tree2_leaf);  // tree2's root becomes child of tree1_leaf

// Iterate over tree
for node in root.iter_depth_first() {
    println!("{}", node.text().unwrap_or_default());
}

// Get all leaves
let leaves = root.iter_leaves();

// Count nodes
let count = root.node_count();

Cost Tracking (OpenRouter)

use minillmlib::{ChatNode, GeneratorInfo, NodeCompletionParameters, CostInfo};
use std::sync::{Arc, Mutex};

let generator = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");

// Track costs across multiple requests
let total_cost = Arc::new(Mutex::new(0.0));
let cost_tracker = total_cost.clone();

let params = NodeCompletionParameters::new()
    .with_cost_tracking(true)
    .with_cost_callback(move |info: CostInfo| {
        *cost_tracker.lock().unwrap() += info.cost;
        println!("Request cost: {} credits", info.cost);
        println!("Tokens: {} prompt, {} completion", 
            info.prompt_tokens, info.completion_tokens);
    });

let root = ChatNode::root("You are helpful.");
let user = root.add_user("Hello!");
let response = user.complete(&generator, Some(&params)).await?;

println!("Total spent: {} credits", *total_cost.lock().unwrap());

Estimating a call's cost

Cost tracking tells you what a call did cost. Estimation tells you what it will cost, before you send it, so you can decide whether to allow it. Enable the estimate feature.

The GeneratorInfo you already have answers it directly:

use minillmlib::{ChatNode, CompletionParameters, GeneratorInfo};

let generator = GeneratorInfo::openrouter("anthropic/claude-haiku-4.5");
let params = CompletionParameters::new().with_max_tokens(1024);

let root = ChatNode::root("You are terse.");
let prompt = root.add_user("Name three primary colours.");

let usd = generator.estimate_cost_usd(&prompt.thread(), &params).await?;
println!("this call will cost at most ${usd:.6}");

The prices come from OpenRouter's catalog, the one public price sheet, so the model is looked up by its OpenRouter id. An OpenRouter generator's model id already is one. A direct-vendor generator whose id differs (Anthropic's claude-haiku-4-5-20251001 is the catalog's anthropic/claude-haiku-4.5) sets .with_openrouter_name("anthropic/claude-haiku-4.5"): that is what unlocks estimation. The only failure is a model the catalog does not know at all.

One model is served by many providers at different prices. The rates used are the serving provider's own when the generator knows who that is (the built-in Anthropic and OpenAI providers do); otherwise the dearest rate any provider of the model charges, the only figure that is a ceiling wherever routing lands.

The figure is a deliberately high estimate, never a best guess, because only the low side lets you overspend. Every assumption leans expensive; it is still an estimate (tokenizers differ across model families), so treat it as a strong ceiling to reserve against, not a guarantee. Concretely it assumes:

no prompt caching (caching only ever makes the real cost lower);
the largest completion the request permits, including any reasoning budget, which providers bill on top of max_tokens rather than inside it;
a minute of media, for a clip whose length you did not state. If your prompt carries audio or video, set the clip's real duration (with_duration): the one-minute assumption overshoots short clips by a lot (measured live: 5-7x on an 8-second clip) and undershoots anything longer than a minute, which is the one way the estimate can come in below the real cost.

There is no error case. A prompt counted larger than the model accepts is priced as the largest input that model does accept, so you always get a number and never have to handle "unknown". Replace it with the real cost once the call returns.

To sharpen the estimate:

If your OpenRouter routing settings pin one provider (a single-entry order with fallbacks off), ProviderSettings::billing_provider() gives you its slug; price with generator.model_rates_served_by(Some(&slug)) to get exactly that provider's rates.
Pass a clip's length with AudioData::with_duration / VideoData::with_duration and the estimate stops guessing. Audio is billed by the second, at up to a thousand times the text rate on some models, so this matters.

Keep your generators alive. Each GeneratorInfo caches the prices it fetches for an hour, and clones share the cache, so reusing a generator means only the first estimate in an hour touches the network; recreating it per call refetches the price sheet every time. The library holds no registry: if you multiplex many models, pool your generators yourself in a map keyed by generator.pricing_key() (the catalog model id plus provider slug, exactly the pair that determines the price). If you never estimate costs, none of this matters.

API Reference

Core Types

Type	Description
`ChatNode`	A node in the conversation tree
`GeneratorInfo`	LLM provider configuration
`CompletionParameters`	Generation parameters (temperature, max_tokens, etc.)
`NodeCompletionParameters`	Per-request settings (retry, JSON parsing, cost tracking, etc.)
`Message`	A single message with role and content
`MessageContent`	Text or multimodal content
`ThreadData`	Serializable conversation thread with format kwargs
`CostInfo`	Cost and token usage information from completions
`CostResolution`	Whether a reported cost is `Resolved`, `Unpriced`, or `Unknown`

GeneratorInfo Methods

// Pre-configured providers
GeneratorInfo::openrouter(model)         // OpenRouter (OpenAI wire, native USD cost)
GeneratorInfo::openai(model)             // OpenAI (token-only; price via with_token_price)
GeneratorInfo::anthropic(model)          // Native Anthropic /v1/messages, x-api-key auth
GeneratorInfo::claude_subscription(model)// Anthropic wire, Claude Pro/Max OAuth token
GeneratorInfo::custom(name, url, model)  // Custom OpenAI-compatible endpoint

// Auth builder methods
.with_api_key(key)                       // provider chooses header (Bearer / x-api-key)
.with_api_key_from_env("ENV_VAR")
.with_bearer_token(token)                // OAuth / subscription bearer token
.with_bearer_token_from_env("ENV_VAR")

// Other builder methods
.with_token_price(TokenPrice::new(in_per_mtok, out_per_mtok)) // cost estimate for token-only providers
.with_provider(Arc::new(MyProvider))     // swap the wire dialect
.with_header(name, value)
.with_vision()
.with_audio()
.with_max_context(length)
.with_default_params(params)

Claude Subscription (use your Pro/Max plan)

A Claude Pro/Max subscription OAuth token authenticates against the native Anthropic API the same way an API key does, but draws on your subscription's rolling quota (the 5-hour / 7-day window) instead of pay-as-you-go API billing.

claude_subscription resolves the token in this order:

the ANTHROPIC_AUTH_TOKEN env var, if set (explicit override, e.g. from claude setup-token; you keep it fresh);
otherwise the live Claude Code credential at ~/.claude/.credentials.json (claudeAiOauth.accessToken), which Claude Code keeps refreshed, so if you're logged into Claude Code with your subscription, it just works.

use minillmlib::{ChatNode, GeneratorInfo, TokenPrice};

// Anthropic returns token counts but no dollar cost, so set a price for a
// resolved cost ESTIMATE (otherwise tracking reports `Unpriced`).
let generator = GeneratorInfo::claude_subscription("claude-haiku-4-5")
    .with_token_price(TokenPrice::new(1.0, 5.0)); // $/Mtok in, $/Mtok out

let root = ChatNode::root("You are helpful.");
let response = root.chat("Hello!", &generator).await?;

Subscription vs Console. A subscription token (from Claude Code) bills your Pro/Max plan. A Console/API OAuth token (e.g. from the ant CLI) bills your API account, not the subscription; for Console use an API key via GeneratorInfo::anthropic(model). Verify which bucket you're hitting by the response's rate-limit headers: subscription returns anthropic-ratelimit-unified-5h-*; the API tier returns anthropic-ratelimit-input-tokens-limit.

CompletionParameters

Parameter	Type	Default	Description
`max_tokens`	`Option<u32>`	`4096`	Maximum tokens to generate
`temperature`	`Option<f32>`	`0.7`	Sampling temperature
`top_p`	`Option<f32>`	`None`	Nucleus sampling
`top_k`	`Option<u32>`	`None`	Top-k sampling
`stop`	`Option<Vec<String>>`	`None`	Stop sequences
`seed`	`Option<u64>`	`None`	Random seed
`response_format`	`Option<ResponseFormat>`	`None`	Force JSON output
`reasoning`	`Option<ReasoningConfig>`	`None`	Extended-thinking effort/budget
`extra`	`Option<HashMap>`	`None`	Provider-specific keys (incl. OpenRouter routing)

NodeCompletionParameters

Parameter	Type	Default	Description
`system_prompt`	`Option<String>`	`None`	Override system prompt
`parse_json`	`bool`	`false`	Parse/repair JSON response
`force_prepend`	`Option<String>`	`None`	Force response prefix
`retry`	`u32`	`4`	Retry attempts
`exp_back_off`	`bool`	`false`	Exponential backoff
`back_off_time`	`f64`	`1.0`	Initial backoff (seconds)
`max_back_off`	`f64`	`15.0`	Max backoff (seconds)
`crash_on_refusal`	`bool`	`false`	Error if no JSON
`crash_on_empty_response`	`bool`	`false`	Error if empty
`track_cost`	`bool`	`false`	Request and report usage/cost
`token_price`	`Option<TokenPrice>`	`None`	Per-request price override (token-only providers)
`cost_callback`	`Option<CostCallback>`	`None`	Callback for cost info

ProviderSettings (OpenRouter)

Parameter	Description
`order`	Ordered list of providers to try
`sort`	Sort by: "price", "throughput", "latency"
`ignore`	Providers to exclude
`data_collection`	"allow" or "deny"
`allow_fallbacks`	Allow fallback providers

CLI Tool

The library includes a CLI for JSON repair:

# Repair JSON from file
minillmlib-cli input.json

# Repair JSON from stdin
echo '{"key": "value",}' | minillmlib-cli

Running Tests

# Default: all offline tests (unit + offline integration). No API calls, free.
cargo test

# Unit tests only (fast)
cargo test --lib

# Live integration tests (REAL, billed API calls): opt in with the `live` feature.
# Reads OPENROUTER_API_KEY, ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN from the env
# (or a .env); each live test skips gracefully if its key is absent.
cargo test --features live

# Run with output
cargo test -- --nocapture

Without --features live, every network test skips, so cargo test is free, offline, and deterministic even when real keys are present in your environment.

License

MIT License - see LICENSE for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

minillmlib 0.5.7