Skip to main content

ProviderStreamRequestFn

Type Alias ProviderStreamRequestFn 

Source
pub type ProviderStreamRequestFn = extern "C" fn(req_json: StbStringRef, emit: Option<ProviderStreamEmitFn>, emit_user_data: *mut c_void, control: *const ProviderStreamControl, completion: Option<ProviderStreamCompletionFn>, completion_user_data: *mut c_void, out: *mut StbString, user_data: *mut c_void) -> i32;
Expand description

A streaming provider-injection fn (for register_streaming_provider).

Same request envelope and final-response contract as ProviderRequestFn, plus a per-chunk channel and the cancellation protocol:

  • req_json — borrowed { model, context, options } envelope (never freed by the plugin).
  • emit + emit_user_data — the host’s per-chunk callback. emit is None only if the host cannot accept chunks; a plugin then degrades to buffering and returning the message through out. While the caller is cancelled the host acknowledges structural cleanup chunks and rejects content chunks (nonzero rc). Cleanup chunks are discarded after the aborted terminal.
  • control — pointer to the host’s ProviderStreamControl, present on every call. It is null only against a host that cannot describe its control surface; a plugin then treats the request as not cancellable and behaves as if cancellation never happens. The plugin must not retain it: it is valid only for the duration of the request fn.
  • completion — reserved; always None on this host (see ProviderStreamCompletionFn). A plugin must return its final message through out and may not expect to keep streaming after returning.
  • out — owning, plugin-owned, plugin-freed response holding the final AssistantMessage JSON (identical to the one-shot contract).
  • user_data — the opaque value the plugin registered with.

Call order: zero or more emit chunks, then exactly one return. The host derives the terminal Done/Error event from out (or from a nonzero rc), so a plugin that emits deltas still must fill out.

Cancellation is opt-in per chunk, not per registration. There is one streaming registration and one signature; whether a plugin checks control is up to it. A plugin that ignores control behaves exactly like the simpler fire-and-forget shape (the host still stops forwarding content and terminates the run on Aborted) — it just keeps generating until it finishes on its own.

Chunk envelope (JSON object with a type discriminator). contentIndex is the index of the block inside the final message; each index may be started once, but blocks may be opened in any order:

{"type":"start"}
{"type":"text_start","contentIndex":0}
{"type":"text_delta","contentIndex":0,"delta":"hello "}
{"type":"text_end","contentIndex":0,"content":"hello world"}
{"type":"thinking_start","contentIndex":1}
{"type":"thinking_delta","contentIndex":1,"delta":"considering"}
{"type":"thinking_end","contentIndex":1,"content":"considering..."}
{"type":"toolcall_start","contentIndex":2,"id":"call_1","name":"echo"}
{"type":"toolcall_delta","contentIndex":2,"delta":"{\"text\""}
{"type":"toolcall_end","contentIndex":2,"id":"call_1","name":"echo","arguments":{}}
{"type":"usage","usage":{"input":1,"output":2}}

Unknown type values are ignored (forward compatibility). A *_delta may carry an optional partial object (a full assistant message) to override the host-side accumulated snapshot; otherwise the host accumulates deltas itself.