Expand description
Ordered response streaming.
The engine emits raw events as soon as they happen: tokens from parallel
LLM nodes interleave, and nodes finish in any order. A chat-style consumer
wants something different: the text of each response node (answer, and
optionally end) rendered in template order, with model tokens streamed
through as early as it is safe to do so.
ResponseAssembler is a consumer-side transducer that does exactly that:
ⓘ
let mut assembler = ResponseAssembler::new(engine.graph());
while let Some(event) = run.next().await {
for chunk in assembler.push(&event) {
print!("{}", chunk.text);
}
}§Rules
- Only response nodes of the root scope are assembled. Events from child
frames (iteration / loop bodies) are ignored, unless
ResponseAssembler::pass_through_unreferencedis enabled, in which case their chunks are forwarded verbatim. - At most one response is active at a time. Responses are activated in the
order they become certain to execute (ties broken by topological
order). Certainty is a linear-time fixpoint over the graph — never a path
enumeration:
- a node that started or finished is certain, and so is the root entry;
- a node is certain when one incoming edge comes from a certain source and
that edge will be taken if the source succeeds: every edge of a
non-branching node (except
fail-branchedges), or the edge matching the handle a finished branch node selected. Node failures are not anticipated, except that an unfinished node with thefail-branchstrategy makes none of its edges certain (either side may be taken).
- While a response is active, its template is drained left to right:
literal text is emitted immediately; a variable produced by a graph node
emits that node’s buffered stream chunks for the variable’s field as they
arrive; once the node finishes, a variable that never streamed is emitted
whole (
Value::to_text). Variables of skipped or failed nodes render as nothing. Variables outside graph nodes (sys.*,conversation.*, …) are resolved from values given viaResponseAssembler::with_value/ResponseAssembler::with_pool, or render as nothing. endnodes do not stream by default (their outputs are structured data, delivered inRunSucceeded). EnableResponseAssembler::include_endto render them as their output values joined by newlines.- Retries and handled failures never mix attempts. A source node whose
policy allows retries (
retry.max_retries > 0) or handles errors (on_errorother thanfail) is not streamed live: its chunks are buffered and emitted only once the node finishes. Anode_retryevent discards the buffered chunks of the failed attempt. On success the final attempt’s text is emitted; after a handleddefault-valuefailure the default output is emitted instead of any partial text; after afail-branchfailure (or an unhandled one) nothing is emitted for the variable. The trade-off: such nodes lose token-by-token streaming in exchange for an answer that always matches the node’s final output. - Ordering after handled failures follows the engine: a non-branch node with
default-valuecontinues along its regular edges, afail-branchnode along itsfail-branchedges, and a branch node withdefault-valuealong the handle named by itsbranchdefault value (no edge if absent). - A terminal run event (
RunSucceeded,RunFailed,RunAborted) flushes everything that can still be rendered, as doesResponseAssembler::finish.
Structs§
- Response
Assembler - Turns raw run events into ordered response text. See the module docs.
- Response
Chunk - A piece of ordered response text.