Skip to main content

Module response

Module response 

Source
Expand description

Ordered response streaming.

The engine emits raw events as soon as they happen: tokens from parallel LLM nodes interleave, and nodes finish in any order. A chat-style consumer wants something different: the text of each response node (answer, and optionally end) rendered in template order, with model tokens streamed through as early as it is safe to do so.

ResponseAssembler is a consumer-side transducer that does exactly that:

ⓘ
let mut assembler = ResponseAssembler::new(engine.graph());
while let Some(event) = run.next().await {
    for chunk in assembler.push(&event) {
        print!("{}", chunk.text);
    }
}

§Rules

  • Only response nodes of the root scope are assembled. Events from child frames (iteration / loop bodies) are ignored, unless ResponseAssembler::pass_through_unreferenced is enabled, in which case their chunks are forwarded verbatim.
  • At most one response is active at a time. Responses are activated in the order they become certain to execute (ties broken by topological order). Certainty is a linear-time fixpoint over the graph — never a path enumeration:
    • a node that started or finished is certain, and so is the root entry;
    • a node is certain when one incoming edge comes from a certain source and that edge will be taken if the source succeeds: every edge of a non-branching node (except fail-branch edges), or the edge matching the handle a finished branch node selected. Node failures are not anticipated, except that an unfinished node with the fail-branch strategy makes none of its edges certain (either side may be taken).
  • While a response is active, its template is drained left to right: literal text is emitted immediately; a variable produced by a graph node emits that node’s buffered stream chunks for the variable’s field as they arrive; once the node finishes, a variable that never streamed is emitted whole (Value::to_text). Variables of skipped or failed nodes render as nothing. Variables outside graph nodes (sys.*, conversation.*, …) are resolved from values given via ResponseAssembler::with_value / ResponseAssembler::with_pool, or render as nothing.
  • end nodes do not stream by default (their outputs are structured data, delivered in RunSucceeded). Enable ResponseAssembler::include_end to render them as their output values joined by newlines.
  • Retries and handled failures never mix attempts. A source node whose policy allows retries (retry.max_retries > 0) or handles errors (on_error other than fail) is not streamed live: its chunks are buffered and emitted only once the node finishes. A node_retry event discards the buffered chunks of the failed attempt. On success the final attempt’s text is emitted; after a handled default-value failure the default output is emitted instead of any partial text; after a fail-branch failure (or an unhandled one) nothing is emitted for the variable. The trade-off: such nodes lose token-by-token streaming in exchange for an answer that always matches the node’s final output.
  • Ordering after handled failures follows the engine: a non-branch node with default-value continues along its regular edges, a fail-branch node along its fail-branch edges, and a branch node with default-value along the handle named by its branch default value (no edge if absent).
  • A terminal run event (RunSucceeded, RunFailed, RunAborted) flushes everything that can still be rendered, as does ResponseAssembler::finish.

Structs§

ResponseAssembler
Turns raw run events into ordered response text. See the module docs.
ResponseChunk
A piece of ordered response text.