Crate onwards 

Onwards - A flexible LLM proxy library

Onwards provides core functionality for building LLM proxy services that can route requests to multiple AI model endpoints with authentication, rate limiting, and request transformation.

Quick Start

use onwards::{AppState, build_router, config::Config, target::Targets};
use axum::serve;
use tokio::net::TcpListener;
use clap::Parser;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Parse CLI configuration
    let config = Config::parse();
     
    // Load targets from configuration file
    let targets = Targets::from_config_file(&config.targets).await?;

    // Create application state with connection pool settings from config
    let app_state = AppState::new(targets);

    // Build router with proxy routes
    let app = build_router(app_state);

    // Start server
    let listener = TcpListener::bind("0.0.0.0:3000").await?;
    serve(listener, app).await?;
    Ok(())
}

Re-exports

pub use traits::NoOpResponseStore;

pub use traits::NoOpToolExecutor;

pub use traits::RequestContext;

pub use traits::ResponseStore;

pub use traits::StoreError;

pub use traits::ToolError;

pub use traits::ToolExecutor;

pub use traits::ToolSchema;

Modules

auth

Authentication utilities for secure API key validation

client

HTTP client abstraction for forwarding requests to upstream services

config

Configuration parsing and validation for the proxy server

errors

Error handling and response structures

handlers

HTTP request handlers for the proxy server

load_balancer

Load balancer for distributing requests across multiple providers

models

Data models for OpenAI-compatible API endpoints

response_sanitizer

Response sanitization module for OpenAI-compatible API responses

sse

SSE (Server-Sent Events) stream buffering

strict

Strict mode router with typed handlers and schema validation

target

Target management and configuration

telemetry

Optional OpenTelemetry tracing initialization for onwards.

test_utils

traits

Extensibility traits for Open Responses adapter

Structs

AppState

The main application state containing the HTTP client and targets configuration

Functions

build_metrics_layer_and_handle

Builds a layer and handle for prometheus metrics collection

build_metrics_router

Builds a router for the metrics endpoint

build_router

Build the main router for the proxy

create_openai_sanitizer

Creates the default OpenAI response sanitization function

extract_model_from_request

Extract the model name from a request

Type Aliases

BodyTransformFn

Type alias for body transformation function

ResponseTransformFn

Type alias for response transformation function