#[non_exhaustive]pub struct StreamTimeouts {
pub setup: Option<Duration>,
pub idle: Option<Duration>,
}Expand description
Time bounds for a streaming request, enforced above the transport.
A streaming response has no total deadline (a long answer is not a failure), but a dead endpoint or a stalled connection must not hang a caller forever. Two bounds catch that:
setup: from sending the request until the response status and headers arrive.idle: the longest wait for a body chunk, the first one included. Only waiting on the transport counts: for each chunk, the clock starts when the reader polls and the chunk is not ready yet, so the time before the reader’s first poll or between its polls does not count.
send enforces both on any HttpTransport; provider
crates call it for their streaming requests (non-streaming requests carry
a whole-exchange HttpRequest::timeout instead). Expiry fails with a
TransportErrorKind::Timeout error, which providers surface as
allama::Error::Timeout, retryable and fallback-eligible.
The defaults (DEFAULT_SETUP,
DEFAULT_IDLE) are deliberately generous: they
detect dead connections, not slow models. Reasoning models can stay
silent for minutes before their first token, and an idle bound that cuts
such a stream off bills the reasoning and returns nothing. Tighter,
workload-specific bounds belong in allama::wrappers::TimeoutProvider,
which wraps the decoded stream and fires first when its limits are
shorter. It can also bound a stream’s total duration
(TimeoutPolicy::stream_total), which this transport layer leaves to
the application.
Timers come from futures-timer (the same timers TimeoutProvider
uses), so the bounds work on any async runtime and, through the browser
or worker setTimeout, on wasm32-unknown-unknown.
use std::time::Duration;
use allama_http::StreamTimeouts;
let tight = StreamTimeouts::default().idle(Some(Duration::from_secs(30)));
assert_eq!(tight.setup, Some(StreamTimeouts::DEFAULT_SETUP));
// A local server that may load a model first: no bounds at all.
let local = StreamTimeouts::unbounded();
assert_eq!((local.setup, local.idle), (None, None));Fields (Non-exhaustive)§
This struct is marked as non-exhaustive
Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.setup: Option<Duration>Bound on the wait for the response status and headers.
idle: Option<Duration>Bound on the wait for each body chunk, the first one included, counted from the reader’s poll that finds the chunk not ready.
Implementations§
Source§impl StreamTimeouts
impl StreamTimeouts
Sourcepub const DEFAULT_SETUP: Duration
pub const DEFAULT_SETUP: Duration
Default setup bound: 120 s, the same budget the
provider crates give a whole non-streaming call. A streaming endpoint
normally answers with headers right away and keeps the model’s
latency for the body.
Sourcepub const DEFAULT_IDLE: Duration
pub const DEFAULT_IDLE: Duration
Default idle bound: 300 s. Longer than a non-streaming
call’s whole default budget on purpose: streaming is how callers reach
reasoning models that think for minutes before their first token, and
not every endpoint sends keep-alive events meanwhile.
Sourcepub const fn unbounded() -> Self
pub const fn unbounded() -> Self
No bounds: the stream may wait forever (for example for a local server that loads the model before answering).
Sourcepub async fn send<T>(
self,
transport: &T,
request: HttpRequest,
) -> Result<HttpResponse, TransportError>where
T: HttpTransport + ?Sized,
pub async fn send<T>(
self,
transport: &T,
request: HttpRequest,
) -> Result<HttpResponse, TransportError>where
T: HttpTransport + ?Sized,
Send request through transport with these bounds: the response
headers must arrive within setup, and its body then fails when no
chunk arrives within idle.
Expiry drops the transport’s future or body, cancelling the request.
§Errors
The transport’s errors, or a TransportErrorKind::Timeout error
when the setup bound expires. An idle expiry is the body’s last item.