pub type NemoRelayNativeAsyncStreamMiddlewareCb = unsafe extern "C" fn(user_data: *mut c_void, invocation_json: *const NemoRelayNativeString, next: *const NemoRelayNativeAsyncNext, stream: *const NemoRelayNativeAsyncStream) -> u32;Expand description
Incremental native LLM stream intercept callback.
The callback owns next and stream and must release each exactly once.
It may push chunks before returning or retain the handles and return
Pending; no implicit timeout is applied. Relay can invoke separate
middleware calls concurrently without stable OS-thread affinity. Retained
handles may be used from a plugin-owned thread, while callbacks supplied to
async_next_invoke_stream run on a Relay runtime worker. The output stream
owns the callback lifetime: next may be invoked repeatedly or concurrently
until that stream finishes, rejects, or is cancelled, and each invocation
has independent callback state. Relay rejects or cancels unfinished and
later invocations after settlement. The plugin must synchronize shared
user_data and callback state and serialize each handle’s final release
after its last operation returns.