cortexkit-rich-ui-syntax

A standalone Rust front end for rich UI language v1, with the reference scanner used before storing tool-output snapshots. It has no host, catalog, network, filesystem-resolution, or rendering dependency.

Parsing and scanning

parse(source, Mode::Reply | Mode::Program) returns a spanned syntax tree and bounded diagnostics. Spans are original UTF-8 byte offsets, start inclusive and end exclusive, including any leading BOM. NUL is replaced in text/string values. parse_bytes repairs invalid UTF-8 first; its offsets describe the repaired text.

The tree includes paragraphs, ATX headings, thematic breaks, fences, quotes, lists/task markers, tables, emphasis/strong/strikethrough, code spans, links and image-as-link syntax. Indented code, setext headings, raw HTML, reference-style links, footnotes, math and front matter are not additional syntax. Tags have named props, inline or block children, and source spans. Program expressions include exact decimal literals, strings/escapes, references, literal-only Artifact/File handles, names/state, calls, lists/ordered records, grouping, postfix lookup/indexing, unary/binary operators and conditionals. Program interpolations and declarations use the same expression parser.

This is a syntax front end, not a component/type checker or evaluator. Addresses in link nodes are inert syntax, not authorization to open or fetch a URL; a host must validate/canonicalize addresses and enforce its network policy. Evaluation limits on values, actions, sites, allocation, fuel and rendered trees belong to that later layer, not to source parsing.

use cortexkit_rich_ui_syntax::{check_token_slot, scan_references, Mode, DIALECT};

assert_eq!(DIALECT, "rich-ui/1");
let scan = scan_references("See $$tc(7). `$$tc(8)`", Mode::Reply);
assert_eq!(scan.references.len(), 1);
assert_eq!(scan.references[0].n, "7");
assert_eq!((scan.references[0].span.start, scan.references[0].span.end), (4, 11));
assert!(check_token_slot(" $$tc(7)").is_none());

scan_references calls parse and walks only reference nodes in that tree. It returns occurrences in source order, preserving repeats, with no regex scan, substitution, number resolution, or parsing of data strings. N is canonical decimal stored as a string, so it cannot silently overflow a machine integer.

Reference rule

$$tc(N) has no internal whitespace or leading zeros; 0 is legal. It counts only where the syntax parser produces a reference node. In particular:

• Markdown prose, tag children, expression props and program expressions count.
• Fences, code spans, escaped markers and expression/prop string literals do not.
• A reply tag with a brace value outside the reply-value grammar becomes one opaque literal span, including its children. No reference inside survives. Reply values are literals, references, literal handles, lists/records of reply values, and Parse(reference, string[, literal-record]); no arbitrary calls, operators, names, state, grouping, or nested components are admitted as props.
• Crossed/missing closes invalidate the enclosing tag span. A child's forbidden prop is local: it does not invalidate a well-formed parent or its references.
• Unknown component names are a catalog matter. Valid reply values on an unknown component still count here. Runners must not substitute a catalog filter for this scanner.
• There is no recognition by decoding entities and then scanning the result.

Runner integration

Only scan the source fields declared as reference slots by the tool provider; never scan arbitrary arguments, tool output, snapshot data, or nested strings.

For every returned occurrence, bind the number to the immutable request that produced the source, not to a later numbering. Store dialect and original byte span with the record. This crate never looks up or copies tool output.

Occurrences after the first 64 have excess = true: record them as unavailable (too_many_refs), never materialize snapshots for them. The caller must also apply the 64-occurrence budget across multiple source/token slots in one call or message, and enforce its separate snapshot-byte budget. fallback means the whole parsed prefix became bounded literal text and has no reference nodes; shortened means the unparsed tail must not be scanned by another implementation. Program diagnostics require fallback before execution; the syntax tree is not an execution approval.

Bounds and streaming

Programs over 64 KiB are refused before parsing. Replies stop at a UTF-8 boundary within 256 KiB. Parsing budgets are 8,192 AST allocations, 2,048 blocks, 65,536 text scalars, 64 reply tags, 256 declarations, expression nesting 64, tag nesting 32, Markdown quote/list nesting 16, emphasis nesting 64, and 50 listed diagnostics (with a separate suppressed count). Reference and handle record budgets are 64 each; excess occurrences remain syntax rather than silently disappearing.

The size limits measure raw input: parse_bytes cuts at the byte limit before UTF-8 repair, so repair (which can triple an invalid byte) never makes a source within the limit too large, and its repaired text, which offsets describe, may be longer than the limit: at most three times it, since each invalid byte grows to at most three. A caller that budgets by text size should charge the repaired text's size, not the input's.

Streaming replies

StreamParser::new(mode) accepts a source in arbitrary byte chunks:

use cortexkit_rich_ui_syntax::{parse, Display, Mode, StreamParser};

let source = "Look at `<Chart/>` and $$tc(3).";
let mut stream = StreamParser::new(Mode::Reply);
let mut drawn = None;
for chunk in source.as_bytes().chunks(5) {
    if let Some(revision) = stream.feed(chunk) {
        drawn = Some(revision); // replace whatever was drawn before
    }
    let work = stream.work(); // charge work.scanned + work.reexamined
    assert!(work.scanned <= chunk.len());
}
let last = drawn.unwrap();
assert!(matches!(last.nodes[0].children[1].kind, Display::Code(_)));
assert_eq!(stream.finish(), parse(source, Mode::Reply));

Its rules:

• Equivalence. finish() returns exactly parse_bytes of everything received, for every chunking: tree, references, diagnostics and spans. The parser keeps the received bytes (up to the limit plus three bytes) and parses them once, at finish. The final result never comes from provisional state.

• Revisions. In reply mode, feed returns a Revision whenever the drawable picture of the received prefix differs from the last one it returned, and None otherwise; revision() gives the current one. Programs are not drawn: in program mode feed always returns None.

• Stable per prefix. A revision is a pure function of the received prefix. The same bytes give an equal revision however they were chunked.

• Not monotone. A later byte may reclassify earlier provisional text: a closing backtick or emphasis delimiter, a link destination, a table delimiter row under a drawn line, or the rest of a line after a tag. A caller drawing revisions replaces the drawn revision with each new one.

• Inert. A revision carries display structure only. References are Display::PendingReference placeholders without their number, links and images carry their label but no destination, and tags carry their name but no props. Nothing in a revision names data to fetch or an action to take, and a revision is never a storage-time reference decision.

• Linear work. A provisional machine consumes each byte once into bounded block and inline state; it never reparses the received prefix. work() reports, for the last feed, the bytes scanned and the held bytes reexamined. Fuel accounting stays with the caller, which reserves part of a reply's budget for the final parse and charges provisional work separately.

• Held context and caps. Text whose classification is still open is held, not drawn: a code span until its closing backticks, a link until its destination closes, a tag until it is complete (a complete tag inside an unresolved code span or link is not complete for display), a possible $$tc(N), an escape, and a line start that may still become a block marker. A paragraph's first line containing | is drawn as paragraph text, and a copy is kept until the next line arrives; when that line is a matching delimiter row (which is held, not drawn), the first line is redrawn as the table header. Each held region has its own cap, all within the 256 KiB reply limit:

  Held region	Cap
  code span, from its opening backticks	STREAM_CONTEXT_BYTES: 32 KiB, in bytes
  link or image, from [ or ![ (labels also nest at most 4 brackets deep)	32 KiB, in bytes
  tag opening, from < to its >	TAG_OPENING_BYTES: 4 KiB, in bytes
  whole tag, from < to the end of its closing tag	32 KiB, in bytes
  possible reference, from $	PENDING_REFERENCE_CHARS: 24 characters
  thematic-break candidate line, fence info string	32 KiB (ASCII marks, so bytes and characters agree; the info string counts bytes)
  kept copy of a possible table header line	32 KiB, in bytes

  At a cap the held text is drawn as a provisional Display::Literal with control characters replaced, and the machine continues from its bounded state after it; a table header copy over its cap is simply dropped, so that table is drawn as a paragraph until the final parse. The caps only shape the provisional picture: the final parse has none of them, and its own limits are the same for every chunking.

• Block ends. A blank line, or the end of a block, resolves what it holds. An unclosed code span or link has its opening marker drawn as text and the rest examined once more; whatever that pass would hold again is drawn without a further pass (as text, or a tag as a Display::Literal). A held escape, !, <, delimiter run or possible reference is drawn as text. A blank line inside a tag's opening or an inline tag's body ends the tag, which the final parser then classifies (usually as literal text); a block tag's body continues. Reply declarations and prose braces are never held: they are drawn as literals as they arrive and end where the final parser ends them (the declaration's last line, the matching } or a blank line).

The provisional machine draws paragraphs, ATX headings, thematic breaks, fenced code, quotes, lists (with an ordered list's start number), GFM tables, emphasis, code spans, links and images, hard and soft breaks, reply prose braces and declarations as literals, complete tags and reference placeholders. Task list markers are drawn as text ([ ] task), except inside a completed tag's body, which the final parser classifies. Completed tags and links are classified by running the final parser over just their held text. Its picture is a preview: where it differs from the final tree, the final tree is authoritative.

All display text replaces control characters other than line feed and tab with U+FFFD; carriage returns become line feeds. That is a floor, not a guarantee that the text is safe to display: a caller drawing it still applies its own display sanitizing. Display spans are byte offsets in the received text after the UTF-8 repair parse_bytes applies: each maximal invalid byte subsequence becomes one U+FFFD. A possibly incomplete scalar at the end of the received bytes is held until the next byte decides it.

Choices

These choices resolve underspecified boundaries conservatively; they are part of this scanner's pinned behavior, not catalog rules:

1. A failed opening candidate is not a tag. Its prefix is opaque through the next > outside strings/braces, the first blank line, or end of source, whichever comes first. Parsing resumes as Markdown after that boundary. This prevents a blank line in an unfinished opener from consuming later paragraphs, without copying output mentioned in the failed prefix. A valid but forbidden reply tag instead makes its whole paired/self-closing span literal.
2. Reply declarations (including continuation lines) and brace-delimited prose are opaque literal text, so references in program examples are not copied. Unclosed prose braces stop at a blank line/end. Ordinary quotation marks in prose are not expression strings; use code spans/fences for quoted programs. This also means braces in a tag written as text inside a link label stay literal in reply mode.
3. Link labels/alt text are inline content, but never components. Destinations, titles and HTTP(S) autolink addresses are not visible reference nodes. Markdown body references count in program mode too, not just declaration expressions.
4. Parenthesized reply values are rejected, even around a reference: grouping is not in the reply-value grammar. Parse options are literal records; their literal lists contain scalars only, as specified by that narrower grammar.
5. There is no maximum numeric value for reference N. The pending-reference streaming threshold is not a final token-length limit. Exact-token slots do have the same 256 KiB hard input ceiling as reply sources, to bound allocation.
6. Work and stack safety apply to all modes: the AST/block budgets also protect programs; mixed expression/tag parsing shares 65 active expression frames; recursive inline contexts cap at 64. A deterministic work allowance of max(1024, 128 * bounded_source_bytes) prevents repeated failed lookaheads from making adversarial parsing superlinear. Exhaustion selects literal fallback or a literal failed construct, never extra references. These extra safety bounds can reject deeply mixed constructs before their independent expression/tag limits would be reached.
7. Diagnostics carry a subject where the failure has a name: reserved and duplicate declarations, duplicate props and record keys (children when a tag sets that prop and also has children), unclosed and mismatched tags (the opening tag's name), forbidden reply props, non-literal state initializers and handles, the tag that passes the reply tag limit (E_LIMIT_TAGS), and a tag nested too deep (E_NESTING). Expression nesting, parse-work, size and count limits have no subject. String diagnostics span the string from its opening quote to where it broke (E_UNTERMINATED_STRING) or the escape as far as it was read (E_BAD_ESCAPE); other diagnostics keep the span of their construct. A string that reaches the end of its construct without a closing quote is unterminated, like one that reaches a line break.

The original 27-case sample file is byte-identical and SHA-256 pinned in tests. refs-scan-samples-v1.1.json adds failed-tag, literal-boundary, link, Unicode and program-mode cases. refs-scan-samples-v1.2.json adds streaming hazards and the string, reserved-name and subject diagnostics; its cases may also pin the exact diagnostics list. Tests load all three from CARGO_MANIFEST_DIR; no checkout layout or platform-specific service is required. Generated boundary/adversarial tests cover parser budgets and randomized/truncated UTF-8 source, and every sample is checked under every split position, one-byte chunks and adversarial chunkings.

The sample files ship in test-vectors/rich-ui-syntax-v1/. With the test-vectors feature, the public vectors module exposes their contents and SHA256SUMS, so callers can test against the exact same bytes without a repository checkout.

Development

cargo test
cargo clippy --all-targets -- -D warnings
cargo fmt --check

The pinned pulldown-cmark dependency supplies CommonMark delimiter behavior, GFM tables/task/strike syntax, entity decoding and source offsets. A byte-length preserving shielding layer keeps rich UI syntax and failed literals opaque to Markdown. serde_json (sample loading) and sha2 (immutable fixture digest) are test-only dependencies. The workspace lockfile pins the exact dependency graph; the crate inherits common package metadata from the workspace.