foundry-local 0.1.0

SDK for Microsoft Foundry Local service
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

# Foundry Local Rust SDK


A Rust SDK for interacting with the Microsoft Foundry Local service. This SDK allows you to manage and use AI models locally on your device.  See [Foundry Local](http://aka.ms/foundrylocal) for more infromation.

## Features

- Start and manage the Foundry Local service
- Download models from the Foundry catalog
- Load and unload models
- List available, cached, and loaded models
- Interact with loaded models using a simple API

## Usage


```rust
use foundry_local::FoundryLocalManager;
use anyhow::Result;

#[tokio::main]

async fn main() -> Result<()> {
    // Create a FoundryLocalManager instance with the option to automatically download and start the service and a model
    let manager = FoundryLocalManager::builder()
        .alias_or_model_id("phi-3.5-mini")
        .bootstrap(true)
        .build()
        .await?;
    
    // Use the OpenAI compatible API to interact with the model
    let client = reqwest::Client::new();
    let response = client
        .post(format!("{}/chat/completions", manager.endpoint()?))
        .json(&serde_json::json!({
            "model": model_info.id,
            "messages": [{"role": "user", "content": prompt}],
        }))
        .send()
        .await?;
    
    let result = response.json::<serde_json::Value>().await?;
    println!("{}", result["choices"][0]["message"]["content"]);
    
    Ok(())
}
```

## Installation


Add the following to your `Cargo.toml`:

```toml
[dependencies]
foundry-local = "0.1.0"
```

## Requirements


- Foundry Local must be installed.  On Windows you can run the following to install latest.
```powershell
winget install Microsoft.FoundryLocal
```
- Rust 1.70.0 or later

## License


Licensed under the MIT License.