Skip to main content

proxy

Function proxy 

Source
pub fn proxy(
    upstream: &str,
    req: &Request,
    client: &mut TcpStream,
    connect_timeout: Duration,
) -> Result<(), String>
Expand description

Forward a request to a model’s endpoint and relay the response back.

Relayed in chunks rather than buffered, so a streamed completion reaches the caller token by token. Buffering here would hold every token until the model finished — turning a live stream into a long silence followed by a wall of text, which is the single most noticeable way a gateway ruins a chat UI.