openai-api-dispatch

OpenAI-compatible chat requests through memory or NATS queues.

Producers submit typed tasks, workers call the configured API, and responses return through the queue.

NATS quick start

Run a NATS server, then configure the model and OpenAI-compatible endpoint:

export OPENAI_API_KEY=your-key
export OPENAI_API_URL=https://api.openai.com/v1
export OPENAI_API_DEFAULT_MODEL=gpt-4.1-mini
export OPENAI_API_NATS_URL=nats://127.0.0.1:4222

This example starts a worker and sends one call through NATS. Use the default crate features and enable Tokio's macros, rt-multi-thread, and time features.

# #[cfg(feature = "nats-queue")]
use openai_api_dispatch::{
    queue::nats::NatsProducer,
    task::TaskBuilder,
    worker::NatsOpenaiWorker,
};

# #[cfg(feature = "nats-queue")]
#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let worker = NatsOpenaiWorker::from_env_or_default().await?;
    let _worker = tokio::spawn(worker.run());

    let producer = NatsProducer::from_env_or_default().await?;
    let task = TaskBuilder::new()
        .with_prompt("Reply with a one-line greeting")
        .build_chat()?;

    let response = task.send_and_wait(&producer, Some(30)).await?;
    anyhow::ensure!(response.success, "{}", response.contents);
    println!("{}", response.contents);

    Ok(())
}
# #[cfg(not(feature = "nats-queue"))]
# fn main() {}

Workers subscribe to openai-api-queue/<model> by default. Override the prefix with OPENAI_API_NATS_PREFIX. In production, run workers and producers as separate processes.

Configuration and behavior

Environment settings are read when constructing producers, workers, and executors.

• A task's explicit model overrides the default and selects its NATS subject.
• Constructors do not wait for server confirmation of subscriptions, so startup can race with publishing.
• Each worker handles one task at a time.
• Queue/API errors stop its loop without an error reply; supervise spawned workers.
• send_and_wait limits only the reply wait, not submission, and expiry does not cancel execution.
• Check response.success even when the call returns Ok.
• Only non-streaming chat is implemented.
• The current prompt is sent as a user message; system entries in input history are ignored (use with_system).
• Schemas request strict JSON output without local validation.
• payload is caller metadata, not model input.
• no_std generated IDs can repeat after restart or wraparound.

Features

Default features are std, memory-queue, and nats-queue; NATS implies std. The API executor requires std.

no_std

For memory queues, use the feature memory-queue. An allocator and pointer/32-bit atomics are required.

This backend polls once, has no timeout support, and evicts old items when bounded; the std backend uses bounded Tokio channels and backpressure.

Development

Run cargo test for local tests. just check also requires cargo-hack and the thumbv7em-none-eabi target. Live integration tests are ignored by default: run just check-nats MODEL [URL] or just check-openai MODEL [URL] against configured servers.

License

Licensed under either the MIT license or the Apache License, Version 2.0, at your option.