Skip to main content

Module dispatch

Module dispatch 

Source
Expand description

The provider-neutral attempt loop every inference route runs (Go BaseAPIHandler.Execute*WithAuthManager plus Manager.Execute*).

Model to providers through the registry, credential selection across all of them, per-credential alias pools, request-scoped rules, retry rounds and stream bootstrap. Routes only parse their request and render the Done or Failure.

Structs§

Call
One client request, already parsed by its route.
Media
How a media route runs through run: which executor operation serves it, and the same credential binding a SessionTurn has (WithPinnedAuthID, WithSelectedAuthIDCallback), which the video routes use to keep a job on the credential that created it.
RunError
A terminal failure of run. bootstrap marks a stream that failed before its first payload (Go streamBootstrapError).
SessionTurn
How a session transport runs one turn through run (Go ExecuteStreamWithAuthManager with WithPinnedAuthID, WithSelectedAuthIDCallback and the execution session).
SseError
The event: error frame a route attaches to an error response (as a response extension), for serve to write instead when the event stream already committed (writeImagesStreamErrorEvent).
Trace
The trace ID of the last credential selected for a request.

Enums§

Done
Failure
MediaKind

Functions§

jitter
Go jitteredCooldownWait: up to a quarter of the wait (at most 2s) extra, never past the configured maximum, so synchronized clients spread out.
peer
request_id
A UUIDv7 request ID (Go logging.GenerateRequestID).
route_path
Go RequestPathMetadataKey (gin FullPath()): the matched route template in gin’s spelling (/v1beta/models/*action), or the URI path when no route matched.
run
Runs one request through selection, execution and retry rounds (Go Manager.Execute*). Selection failures take part in retry rounds too.
run_with_bootstrap_retries
Go handlers_stream.go: a stream that failed before its first payload is retried as a whole request when the status is statusless, auth, quota, timeout or 5xx.
serve
Runs call and renders the result, adding Go’s X-CPA-TRACE-ID (selection time, the selected credential’s auth index and a request ID) once a credential was picked.
upstream_summary
Go ExtractUpstreamErrorSummary: the upstream error’s code and message, sanitized.

Type Aliases§

OnSelected
The SessionTurn::on_selected callback.
Peer
The downstream peer address, as the listener records it (ConnectInfo).