cortexkit-rich-ui-syntax

A standalone Rust front end for rich UI language v1, with the reference scanner used before storing tool-output snapshots. It has no host, catalog, network, filesystem-resolution, or rendering dependency.

Parsing and scanning

parse(source, Mode::Reply | Mode::Program) returns a spanned syntax tree and bounded diagnostics. Spans are original UTF-8 byte offsets, start inclusive and end exclusive, including any leading BOM. NUL is replaced in text/string values. parse_bytes repairs invalid UTF-8 first; its offsets describe the repaired text.

The tree includes paragraphs, ATX headings, thematic breaks, fences, quotes, lists/task markers, tables, emphasis/strong/strikethrough, code spans, links and image-as-link syntax. Indented code, setext headings, raw HTML, reference-style links, footnotes, math and front matter are not additional syntax. Tags have named props, inline or block children, and source spans. Program expressions include exact decimal literals, strings/escapes, references, literal-only Artifact/File handles, names/state, calls, lists/ordered records, grouping, postfix lookup/indexing, unary/binary operators and conditionals. Program interpolations and declarations use the same expression parser.

This is a syntax front end, not a component/type checker or evaluator. Addresses in link nodes are inert syntax, not authorization to open or fetch a URL; a host must validate/canonicalize addresses and enforce its network policy. Evaluation limits on values, actions, sites, allocation, fuel and rendered trees belong to that later layer, not to source parsing.

use cortexkit_rich_ui_syntax::{check_token_slot, scan_references, Mode, DIALECT};

assert_eq!(DIALECT, "rich-ui/1");
let scan = scan_references("See $$tc(7). `$$tc(8)`", Mode::Reply);
assert_eq!(scan.references.len(), 1);
assert_eq!(scan.references[0].n, "7");
assert_eq!((scan.references[0].span.start, scan.references[0].span.end), (4, 11));
assert!(check_token_slot(" $$tc(7)").is_none());

scan_references calls parse and walks only reference nodes in that tree. It returns occurrences in source order, preserving repeats, with no regex scan, substitution, number resolution, or parsing of data strings. N is canonical decimal stored as a string, so it cannot silently overflow a machine integer.

Reference rule

$$tc(N) has no internal whitespace or leading zeros; 0 is legal. It counts only where the syntax parser produces a reference node. In particular:

• Markdown prose, tag children, expression props and program expressions count.
• Fences, code spans, escaped markers and expression/prop string literals do not.
• A reply tag with a brace value outside the reply-value grammar becomes one opaque literal span, including its children. No reference inside survives. Reply values are literals, references, literal handles, lists/records of reply values, and Parse(reference, string[, literal-record]); no arbitrary calls, operators, names, state, grouping, or nested components are admitted as props.
• Crossed/missing closes invalidate the enclosing tag span. A child's forbidden prop is local: it does not invalidate a well-formed parent or its references.
• Unknown component names are a catalog matter. Valid reply values on an unknown component still count here. Runners must not substitute a catalog filter for this scanner.
• There is no recognition by decoding entities and then scanning the result.

Runner integration

Only scan the source fields declared as reference slots by the tool provider; never scan arbitrary arguments, tool output, snapshot data, or nested strings.

For every returned occurrence, bind the number to the immutable request that produced the source, not to a later numbering. Store dialect and original byte span with the record. This crate never looks up or copies tool output.

Occurrences after the first 64 have excess = true: record them as unavailable (too_many_refs), never materialize snapshots for them. The caller must also apply the 64-occurrence budget across multiple source/token slots in one call or message, and enforce its separate snapshot-byte budget. fallback means the whole parsed prefix became bounded literal text and has no reference nodes; shortened means the unparsed tail must not be scanned by another implementation. Program diagnostics require fallback before execution; the syntax tree is not an execution approval.

Bounds and streaming

Programs over 64 KiB are refused before parsing. Replies stop at a UTF-8 boundary within 256 KiB. Parsing budgets are 8,192 AST allocations, 2,048 blocks, 65,536 text scalars, 64 reply tags, 256 declarations, expression nesting 64, tag nesting 32, Markdown quote/list nesting 16, emphasis nesting 64, and 50 listed diagnostics (with a separate suppressed count). Reference and handle record budgets are 64 each; excess occurrences remain syntax rather than silently disappearing.

StreamParser retains a bounded source prefix plus at most three UTF-8 lookahead bytes, accepts arbitrary byte chunks, and parses once at finalization. It does not expose provisional display frames or provisional reference lists. All split positions and single-byte chunking have the same final result as a whole-source parse, including truncation through a multibyte scalar. A UI needing provisional frames must not treat them as storage-time reference decisions.

Choices

These choices resolve underspecified boundaries conservatively; they are part of this scanner's pinned behavior, not catalog rules:

1. A failed opening candidate is not a tag. Its prefix is opaque through the next > outside strings/braces, the first blank line, or end of source, whichever comes first. Parsing resumes as Markdown after that boundary. This prevents a blank line in an unfinished opener from consuming later paragraphs, without copying output mentioned in the failed prefix. A valid but forbidden reply tag instead makes its whole paired/self-closing span literal.
2. Reply declarations (including continuation lines) and brace-delimited prose are opaque literal text, so references in program examples are not copied. Unclosed prose braces stop at a blank line/end. Ordinary quotation marks in prose are not expression strings; use code spans/fences for quoted programs. This also means braces in a tag written as text inside a link label stay literal in reply mode.
3. Link labels/alt text are inline content, but never components. Destinations, titles and HTTP(S) autolink addresses are not visible reference nodes. Markdown body references count in program mode too, not just declaration expressions.
4. Parenthesized reply values are rejected, even around a reference: grouping is not in the reply-value grammar. Parse options are literal records; their literal lists contain scalars only, as specified by that narrower grammar.
5. There is no maximum numeric value for reference N. The pending-reference streaming threshold is not a final token-length limit. Exact-token slots do have the same 256 KiB hard input ceiling as reply sources, to bound allocation.
6. Work and stack safety apply to all modes: the AST/block budgets also protect programs; mixed expression/tag parsing shares 65 active expression frames; recursive inline contexts cap at 64. A deterministic work allowance of max(1024, 128 * bounded_source_bytes) prevents repeated failed lookaheads from making adversarial parsing superlinear. Exhaustion selects literal fallback or a literal failed construct, never extra references. These extra safety bounds can reject deeply mixed constructs before their independent expression/tag limits would be reached.

The original 27-case sample file is byte-identical and SHA-256 pinned in tests. refs-scan-samples-v1.1.json adds failed-tag, literal-boundary, link, Unicode and program-mode cases. Tests load both from CARGO_MANIFEST_DIR; no checkout layout or platform-specific service is required. Generated boundary/adversarial tests cover parser budgets and randomized/truncated UTF-8 source.

The sample files ship in test-vectors/rich-ui-syntax-v1/. With the test-vectors feature, the public vectors module exposes their contents and SHA256SUMS, so callers can test against the exact same bytes without a repository checkout.

Development

cargo test
cargo clippy --all-targets -- -D warnings
cargo fmt --check

The pinned pulldown-cmark dependency supplies CommonMark delimiter behavior, GFM tables/task/strike syntax, entity decoding and source offsets. A byte-length preserving shielding layer keeps rich UI syntax and failed literals opaque to Markdown. serde_json (sample loading) and sha2 (immutable fixture digest) are test-only dependencies. The workspace lockfile pins the exact dependency graph; the crate inherits common package metadata from the workspace.