Inference Gateway Rust SDK

An SDK written in Rust for the Inference Gateway.

• Inference Gateway Rust SDK
  • Installation
  • Usage
    • Creating a Client
    • Listing Models
    • Listing Models from a specific provider
    • Generating Content
    • Health Check
  • Contributing
  • License

Installation

Run cargo add inference-gateway-sdk.

Usage

Creating a Client

use inference_gateway_sdk::{
    GatewayError,
    InferenceGatewayAPI,
    InferenceGatewayClient,
    Message,
    Provider,
    MessageRole,
};
use log::info;

fn main() -> Result<(), GatewayError> {
    env_logger::init();

    let client = InferenceGatewayClient::new("http://localhost:8080");

    // List available models
    let models = client.list_models()?;
    for provider_models in models {
        info!("Provider: {:?}", provider_models.provider);
        for model in provider_models.models {
            info!("Model: {:?}", model.name);
        }
    }

    let response = client.generate_content(
        Provider::Ollama,
        "llama2",
        vec![Message {
            role: MessageRole::User,
            content: "Tell me a joke".to_string(),
        }],
    )?;

    info!("Response: {:?}", response);
    Ok(())
}

Listing Models

To list all available models from all configured providers, use the list_models method:

use inference_gateway_sdk::{
    GatewayError
    InferenceGatewayAPI,
    InferenceGatewayClient,
    Message,
    Provider,
    MessageRole
};
use log::info;

#[tokio::main]
fn main() -> Result<(), GatewayError> {
    // ...create a client

    // List all providers and models
    let models = client.list_models().await?;
    for provider_models in models {
        info!("Provider: {:?}", provider_models.provider);
        for model in provider_models.models {
            info!("Model: {:?}", model.name);
        }
    }

    // List models for a specific provider
    let resp = client.list_models_by_provider(Provider::Ollama).await?;
    let models = resp.models;
    info!("Provider: {:?}", resp.provider);
    for model in models {
        info!("Model: {:?}", model.name);
    }
}

Listing Models from a specific provider

To list all available models from a specific provider, use the list_models_by_provider method:

// ...rest of the imports
use log::info;

// ...main function
let resp = client.list_models_by_provider(Provider::Ollama).await?;
let models = resp.models;
info!("Provider: {:?}", resp.provider);
for model in models {
    info!("Model: {:?}", model.name);
}

Generating Content

To generate content using a model, use the generate_content method:

// ...rest of the imports
use log::info;

// ...main function
let response = client.generate_content(
    Provider::Ollama,
    "llama2",
    vec![Message {
        role: MessageRole::User,
        content: "Tell me a joke".to_string(),
    }],
).await?;

info!("Provider: {:?}", response.provider);
info!("Response: {:?}", response.response);

Health Check

To check if the Inference Gateway is running, use the health_check method:

// ...rest of the imports
use log::info;

// ...main function
let is_healthy = client.health_check()?;
info!("API is healthy: {}", is_healthy);

Contributing

Please refer to the CONTRIBUTING.md file for information about how to get involved. We welcome issues, questions, and pull requests.

License

This SDK is distributed under the MIT License, see LICENSE for more information.