Foundry Local Rust SDK

A Rust SDK for interacting with the Microsoft Foundry Local service. This SDK allows you to manage and use AI models locally on your device. See Foundry Local for more infromation.

Features

Start and manage the Foundry Local service
Download models from the Foundry catalog
Load and unload models
List available, cached, and loaded models
Interact with loaded models using a simple API

Usage

use foundry_local::FoundryLocalManager;
use anyhow::Result;

#[tokio::main]
async fn main() -> Result<()> {
    // Create a FoundryLocalManager instance with the option to automatically download and start the service and a model
    let manager = FoundryLocalManager::builder()
        .alias_or_model_id("phi-3.5-mini")
        .bootstrap(true)
        .build()
        .await?;
    
    // Use the OpenAI compatible API to interact with the model
    let client = reqwest::Client::new();
    let response = client
        .post(format!("{}/chat/completions", manager.endpoint()?))
        .json(&serde_json::json!({
            "model": model_info.id,
            "messages": [{"role": "user", "content": prompt}],
        }))
        .send()
        .await?;
    
    let result = response.json::<serde_json::Value>().await?;
    println!("{}", result["choices"][0]["message"]["content"]);
    
    Ok(())
}

Installation

Add the following to your Cargo.toml:

[dependencies]

foundry-local = "0.1.0"

Requirements

Foundry Local must be installed. On Windows you can run the following to install latest.

winget install Microsoft.FoundryLocal

Rust 1.70.0 or later

License