ic-llm

A library for making requests to the LLM canister on the Internet Computer.

Supported Models

Models are identified by their string name (passed to ic_llm::prompt and ic_llm::chat). The available models are:

Models are added frequently — see the LLM canister for the authoritative, up-to-date list.

Paying for models

send() automatically attaches 100B cycles to every request. Paid models are charged from those cycles (any unused portion is refunded); free models refund the full amount. Because cycles are always attached, the calling canister must hold at least 100B cycles when send() runs, or the call traps — this applies even when using a free model.

Local Development

When developing locally, the architecture differs slightly from mainnet: instead of relying on AI workers, the LLM canister connects directly to a local Ollama instance. The interface is identical to mainnet — see How Does it Work? for details.

Before running an agent locally, start Ollama and pull the model you intend to use:

ollama serve
ollama run llama3.1:8b   # one-time download

For complete, working project setups — including how to deploy the LLM canister locally — see the examples in this repository (e.g. examples/quickstart-agent-rust).

Usage

Basic Usage

Prompting (Single Message)

The simplest way to interact with a model is by sending a single prompt:

async fn example() -> String {
    ic_llm::prompt("llama3.1:8b", "What's the speed of light?").await
}

Chatting (Multiple Messages)

For more complex interactions, you can send multiple messages in a conversation:

use ic_llm::ChatMessage;

async fn example() {
    ic_llm::chat("llama3.1:8b")
        .with_messages(vec![
            ChatMessage::System {
                content: "You are a helpful assistant".to_string(),
            },
            ChatMessage::User {
                content: "How big is the sun?".to_string(),
            },
        ])
        .send()
        .await;
}

Choosing the LLM canister

By default the SDK addresses the mainnet LLM canister (w36hm-eqaaa-aaaal-qr76a-cai). When your canister is deployed with icp deploy, the SDK transparently picks up PUBLIC_CANISTER_ID:llm if it has been auto-injected — so the same code works against a local llm canister whose principal differs from mainnet, with no caller-side changes.

For other cases (a fork, a mock, a staging deployment under a different name), override the canister explicitly:

use candid::Principal;
async fn example() {
    let custom = Principal::from_text("aaaaa-aa").unwrap();
    ic_llm::chat("llama3.1:8b")
        .with_canister(custom)
        .with_messages(vec![])
        .send()
        .await;
}

Advanced Usage with Tools

Understanding Tools

Tools are custom functions that you define and make available to the LLM. They allow the AI to perform actions beyond just generating text responses. When you provide tools to the LLM, it can decide when and how to use them based on the user's request.

Common use cases for tools:

• Data retrieval: Fetching real-time information (prices, weather, account balances)
• External API calls: Integrating with third-party services
• Calculations: Performing complex computations
• Database operations: Querying or updating data
• Custom business logic: Executing domain-specific functions

How it works:

1. You define available tools with their parameters
2. The LLM analyzes the user's request
3. If a tool would be helpful, the LLM returns a "tool call" instead of a direct answer
4. Your code executes the requested tool with the LLM's provided parameters
5. You send the tool's result back to the LLM
6. The LLM incorporates the result into its final response

Defining and Using a Tool

You can define tools that the LLM can use to perform actions:

use ic_llm::{ChatMessage, ParameterType};

async fn example() {
    ic_llm::chat("llama3.1:8b")
        .with_messages(vec![
            ChatMessage::System {
                content: "You are a helpful assistant".to_string(),
            },
            ChatMessage::User {
                content: "What's the balance of account abc123?".to_string(),
            },
        ])
        .with_tools(vec![
            ic_llm::tool("icp_account_balance")
                .with_description("Lookup the balance of an ICP account")
                .with_parameter(
                    ic_llm::parameter("account", ParameterType::String)
                        .with_description("The ICP account to look up")
                        .is_required()
                )
                .build()
        ])
        .send()
        .await;
}

Handling Tool Calls from the LLM

When the LLM decides to use one of your tools, you can handle the call:

use ic_llm::{ChatMessage, ParameterType, Response};

async fn example() -> Response {
    let response = ic_llm::chat("llama3.1:8b")
        .with_messages(vec![
            ChatMessage::System {
                content: "You are a helpful assistant".to_string(),
            },
            ChatMessage::User {
                content: "What's the weather in San Francisco?".to_string(),
            },
        ])
        .with_tools(vec![
            ic_llm::tool("get_weather")
                .with_description("Get current weather for a location")
                .with_parameter(
                    ic_llm::parameter("location", ParameterType::String)
                        .with_description("The location to get weather for")
                        .is_required()
                )
                .build()
        ])
        .send()
        .await;
    
    // Process tool calls if any
    for tool_call in &response.message.tool_calls {
        match tool_call.function.name.as_str() {
            "get_weather" => {
                // Extract the location parameter
                let location = tool_call.function.get("location").unwrap();
                // Call your weather API or service
                let weather = get_weather(&location).await;
                // You would typically send this information back to the LLM in a follow-up message
            }
            _ => {} // Handle other tool calls
        }
    }
    
    response
}

// Mock function for getting weather
async fn get_weather(location: &str) -> String {
    format!("Weather in {}: Sunny, 72°F", location)
}

Complete Tool Usage Example

Here's a more complete example showing how to handle tool calls and continue the conversation:

use ic_llm::{ChatMessage, ParameterType, Response};

async fn handle_chat_with_tools(user_message: String) -> String {
    let mut messages = vec![
        ChatMessage::System {
            content: "You are a helpful assistant".to_string(),
        },
        ChatMessage::User {
            content: user_message,
        },
    ];

    let tools = vec![
        ic_llm::tool("get_weather")
            .with_description("Get current weather for a location")
            .with_parameter(
                ic_llm::parameter("location", ParameterType::String)
                    .with_description("The location to get weather for")
                    .is_required()
            )
            .build(),
        ic_llm::tool("get_icp_price")
            .with_description("Get the current ICP token price")
            .build()
    ];

    let response = ic_llm::chat("llama3.1:8b")
        .with_messages(messages.clone())
        .with_tools(tools)
        .send()
        .await;

    // Check if LLM wants to use tools
    if !response.message.tool_calls.is_empty() {
        // Add assistant message with tool calls
        messages.push(ChatMessage::Assistant(response.message.clone()));

        // Process each tool call
        for tool_call in &response.message.tool_calls {
            let tool_result = match tool_call.function.name.as_str() {
                "get_weather" => {
                    let location = tool_call.function.get("location").unwrap_or_default();
                    get_weather(&location).await
                }
                "get_icp_price" => {
                    get_icp_price().await
                }
                _ => format!("Unknown tool: {}", tool_call.function.name)
            };

            // Add tool result to conversation
            messages.push(ChatMessage::Tool {
                content: tool_result,
                tool_call_id: tool_call.id.clone(),
            });
        }

        // Get final response from LLM with tool results
        let final_response = ic_llm::chat("llama3.1:8b")
            .with_messages(messages)
            .send()
            .await;

        final_response.message.content.unwrap_or_default()
    } else {
        // No tool calls needed, return direct response
        response.message.content.unwrap_or_default()
    }
}

// Example tool implementations
async fn get_weather(location: &str) -> String {
    // In a real implementation, you would call a weather API
    format!("Weather in {}: Sunny, 72°F", location)
}

async fn get_icp_price() -> String {
    // In a real implementation, you would call a price API
    "Current ICP price: $10.50".to_string()
}