Skip to main content

execute_streaming_with_timeouts

Function execute_streaming_with_timeouts 

Source
pub async fn execute_streaming_with_timeouts(
    catalog: &Catalog,
    route_name: &str,
    legs: &[ChainLeg],
    req: &ChatRequest,
    t: StreamTimeouts,
) -> Result<CommittedStream, GatewayError>
Expand description

Streaming executor with explicit first-chunk timeout. For each leg, start the stream and peek the first item within t.first_chunk. First leg to yield an item is committed; failures (including timeout) before the first item fall back. After commitment there is no fallback.