Crate openai_orch

A concurrency-included Rust client for the OpenAI API.

Overview

openai-orch is designed to provide a simple interface for sending requests to OpenAI in bulk, while managing concurrency at a global level. It also provides configurable policies to control how concurrency, timeouts, and retries are handled.

Usage

To use this library, create an Orchestrator with the desired policies and keys. To allow a thread to use the Orchestrator, simply clone it. To send a request, call add_request on the Orchestrator, and then call get_response on the Orchestrator with the request ID returned by add_request. The Orchestrator will handle concurrency automatically.

Example

use openai_orch::prelude::*;
 
#[tokio::main]
async fn main() {
  let policies = Policies::default();
  let keys = Keys::from_env().unwrap();
  let orchestrator = Orchestrator::new(policies, keys);
 
  let request = ChatSisoRequest::new(
    "You are a helpful assistant.".to_string(),
    "What are you?".to_string(),
    Default::default(),
  );
  let request_id = orchestrator.add_request(request).await;
 
  let response = orchestrator
    .get_response::<ChatSisoResponse>(request_id)
    .await;
  println!("{}", response.unwrap());
}

If you’d like, you can implement OrchRequest on your own request type. See the OrchRequest trait for more information. Currently the only request type implemented is ChatSisoRequest; SISO stands for “Single Input Single Output”.

Modules

chat

Requests and responses using Chat models.

embed

Requests and responses using Embeddings models.

keys

A Keys struct for managing OpenAI keys.

policies

Policies for controlling retry, concurrency, and timeout behavior.

prelude

Provides a useful collection of openai-orch types

utils

Utilites for use when writing custom requests.

Structs

Orchestrator

The central interface for openai_orch. The Orchestrator is responsible for managing the concurrency of requests and their responses.

RequestID

A unique identifier for a request.

Traits

OrchRequest

Allows a request type to be used with the Orchestrator.

ResponseType