Expand description
The streaming chat handler (chat_stream).
Why: the OpenRouter/Ollama tool-calling loop is by far the largest single
concern in the chat surface; isolating it keeps the other handlers readable
(split out of the former monolithic chat.rs, issue #607).
Why it streams at all (#6286): this handler used to answer
POST /api/v1/chat with Content-Type: text/event-stream, pushing LLM
tokens off a ReceiverStream as the model produced them. ADR-0032 retired
that listener, and a one-frame-per-connection JSON-RPC call cannot carry a
token stream — so trusty_common::uds::server’s multi-frame extension
landed for exactly this method rather than chat degrading to a single
buffered blob. The producer shape is unchanged: a background task fills an
mpsc::Sender and this function hands back the receiver.
What: chat_stream, registered as memory.chat with
RpcRouter::typed_stream. Each data: {…} line the SSE version wrote is
now one "stream":"item" frame carrying the same object — {session_id},
{delta}, {tool_call}, {tool_result}. data: [DONE] becomes the
stream’s terminal end frame.
A mid-stream provider failure is the terminal ERROR frame, not an item.
The SSE version wrote data: {"error": …} and then ended normally, which a
reader could not tell from a completed answer — the Fail-Open branch
rpc_chat_reports_a_provider_failure_as_the_terminal_error_frame closes.
Tool-loop building blocks (all_tools, execute_tool, MAX_TOOL_ROUNDS,
ChatBody) come from the sibling tools submodule.
Test: crate::transport::uds::tests — rpc_chat_*.
Functions§
- chat_
stream - Open a
memory.chatstream (#6286).