Skip to main content

Module marker

Module marker 

Source
Expand description

Tail-buffered marker detector for the streaming chat-adapter parser.

Scans incoming text chunks for [[ ## <name> ## ]] patterns. The tricky case is markers split across token boundaries: a chunk might end with [[ ## and the next chunk start with fname ## ]]. The detector holds an internal tail buffer of ambiguous trailing bytes that could be the start of a marker until either the marker completes or enough non-marker content arrives to confirm the bytes are literal content (e.g. a string field happens to contain [[).

Used by super::parser::ChatStreamParser; not part of the public adapter API.