Inference Gateway Rust SDK

An SDK written in Rust for the Inference Gateway.

• Inference Gateway Rust SDK
  • Installation
  • Usage
    • Creating a Client
    • Listing Models
    • Generating Content
    • Health Check
  • License

Installation

Run cargo add inference-gateway-sdk.

Usage

Creating a Client

use inference_gateway_sdk::{InferenceGatewayClient, Message, Provider};
use log::info;
use std::error::Error;

fn main() -> Result<(), Box<dyn Error>> {
    env_logger::init();

    let client = InferenceGatewayClient::new("http://localhost:8080");

    // List available models
    let models = client.list_models()?;
    for provider_models in models {
        info!("Provider: {:?}", provider_models.provider);
        for model in provider_models.models {
            info!("Model: {:?}", model.id);
        }
    }

    let response = client.generate_content(
        Provider::Ollama,
        "llama2",
        vec![Message {
            role: "user".to_string(),
            content: "Tell me a joke".to_string(),
        }],
    )?;

    info!("Response: {:?}", response);
    Ok(())
}

Listing Models

To list available models, use the list_models method:

use log::info;

let models = client.list_models()?;
for provider_models in models {
    info!("Provider: {:?}", provider_models.provider);
    for model in provider_models.models {
        info!("Model: {:?}", model.id);
    }
}

Generating Content

To generate content using a model, use the generate_content method:

use log::info;

let response = client.generate_content(
    Provider::Ollama,
    "llama2",
    vec![Message {
        role: "user".to_string(),
        content: "Tell me a joke".to_string(),
    }],
)?;

info!("Provider: {:?}", response.provider);
info!("Response: {:?}", response.response);

Health Check

To check if the Inference Gateway is running, use the health_check method:

use log::info;

let is_healthy = client.health_check()?;
info!("API is healthy: {}", is_healthy);

License

This SDK is distributed under the MIT License, see LICENSE for more information.