Crate openai_orch

source ·
Expand description

A concurrency-included Rust client for the OpenAI API.

Overview

openai-orch is designed to provide a simple interface for sending requests to OpenAI in bulk, while managing concurrency at a global level. It also provides configurable policies to control how concurrency, timeouts, and retries are handled.

Usage

To use this library, create an Orchestrator with the desired policies and keys. To allow a thread to use the Orchestrator, simply clone it. To send a request, call add_request on the Orchestrator, and then call get_response on the Orchestrator with the request ID returned by add_request. The Orchestrator will handle concurrency automatically.

Example

use openai_orch::prelude::*;
 
#[tokio::main]
async fn main() {
  let policies = Policies::default();
  let keys = Keys::from_env().unwrap();
  let orchestrator = Orchestrator::new(policies, keys);
 
  let request = ChatSisoRequest::new(
    "You are a helpful assistant.".to_string(),
    "What are you?".to_string(),
    Default::default(),
  );
  let request_id = orchestrator.add_request(request).await;
 
  let response = orchestrator
    .get_response::<ChatSisoResponse>(request_id)
    .await;
  println!("{}", response.unwrap());
}

If you’d like, you can implement OrchRequest on your own request type. See the OrchRequest trait for more information. Currently the only request type implemented is ChatSisoRequest; SISO stands for “Single Input Single Output”.

Modules

  • chat
    Requests and responses using Chat models.
  • embed
    Requests and responses using Embeddings models.
  • keys
    A Keys struct for managing OpenAI keys.
  • policies
    Policies for controlling retry, concurrency, and timeout behavior.
  • prelude
    Provides a useful collection of openai-orch types
  • utils
    Utilites for use when writing custom requests.

Structs

  • Orchestrator
    The central interface for openai_orch. The Orchestrator is responsible for managing the concurrency of requests and their responses.
  • RequestID
    A unique identifier for a request.

Traits