Skip to main content

call_inference

Function call_inference 

Source
pub fn call_inference(
    upstream_json: String,
    url: String,
    client: Arc<Client>,
    auth: Option<String>,
    chunk_timeout: Duration,
) -> impl Stream<Item = Result<String, ExecutorError>> + Send + 'static
Expand description

Step 2 — Call the LLM inference backend; yields raw SSE lines (data: …).

Always requests stream=true upstream. Stops on [DONE].

§Errors

Each stream item is Result<String, ExecutorError>. The stream yields Err on: