Struct LlmProxy 

pub struct LlmProxy { /* private fields */ }

The LLM proxy, an Application. See the module docs.

Implementations

impl LlmProxy

pub fn new() -> LlmProxy

pub fn provider(self, provider: Provider) -> LlmProxy

A provider; tried in the order added, per API.

pub fn virtual_key( self, key: impl Into<String>, identity: impl Into<String>, ) -> LlmProxy

A key clients send, and the identity it stands for.

pub fn allow_anonymous(self) -> LlmProxy

Let requests without a known key through, as identity anonymous (a proxy behind its own authentication).

pub fn budget( self, identity: impl Into<String>, tokens: u64, window_secs: u64, ) -> LlmProxy

At most tokens (input + output) per window_secs for identity.

pub fn redis( self, addr: impl Into<String>, password: Option<String>, prefix: impl Into<String>, ) -> LlmProxy

Count budgets in Redis at addr (host:port), under keys starting with prefix, so every replica sees the same spend.

pub fn audit_log(self, on: bool) -> LlmProxy

One JSON line per request (identity, provider, model, status, tokens, time to first byte, duration) through the access log.

pub fn timeouts(self, connect: Duration, read: Duration) -> LlmProxy

How long to wait for a provider’s connection, and for each read of its response (default 10 s and 300 s).

Trait Implementations

impl Application for LlmProxy

fn execute( &self, request: &Request, connection: &ConnectionInfo, ) -> Result<Response, String>

Receives a parsed request and returns a fully-built response. Walk your controller list with is_matching / process and return the first match.

fn runs_inline(&self, _request: &Request, _connection: &ConnectionInfo) -> bool

true when executing request is quick and never waits on anything but the local filesystem — health checks, metrics, small static files. The async servers (HTTP/1.1, HTTP/2, HTTP/3) then call execute directly on the event loop, as nginx does, instead of handing the request to tokio’s blocking pool: that handoff (a shared lock and two cross-thread wake-ups) costs more than serving a small file.

fn execute_async<'a>( &'a self, _request: &'a Request, _connection: &'a ConnectionInfo, ) -> Option<AsyncResponse<'a>>

An asynchronous version of execute for this request, when the application has one: the async servers await it on the connection’s task instead of tying up a blocking-pool thread while it waits (the config-driven proxy uses this to wait for a backend without holding a thread). None (the default) means “use execute”. Wrappers return Some only when they add no synchronous work of their own.

fn streams_body(&self, _request: &Request, _connection: &ConnectionInfo) -> bool

true to receive this request’s body as a stream (Body::Stream) instead of read into memory first: execute then runs while the body is still arriving, on a blocking-pool thread, and reads it with std::io::Read — a multi-gigabyte upload in a few chunks of memory. Asked after the head is read, so request.body is empty here; decide from the method, path and headers.

impl Clone for LlmProxy

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for LlmProxy

fn default() -> Self

Returns the “default value” for a type.