//! Conversion diagnostics: a read-only pass that reports what serializing this
//! AST to that format would silently lose.
//!
//! Twig's serializers degrade or drop a node whenever the target format has no
//! spelling for it — djot's `{=mark=}` written into Markdown comes back as
//! plain text, an HTML comment converted to djot vanishes entirely. None of it
//! is an error (the output is still a valid document), so it happens quietly.
//! `analyze` surfaces it: it walks the tree and returns one `Warning` per lossy
//! node.
//!
//! Ported from fig's `src/diagnostics.zig`, which does the same job for its
//! data formats, and kept deliberately separate from `parse_diagnostic.zig` for
//! the reasons `languages/rst/rst.zig` sets out: a parse diagnostic anchors to a
//! byte span in the SOURCE and is a fact about one document, while a conversion
//! warning has no source offset to point at (the output does not exist yet), so
//! it anchors to a node PATH — and it is a fact about a (document, target)
//! PAIR, which is why it cannot be stored alongside a `Document` at all.
//!
//! ── The table is the gate on the vocabulary ────────────────────────────────
//! `fidelity` switches exhaustively over `Kind` AND over the three family enums
//! (`InlineMark`, `TextLeafKind`, `MarkupLeafKind`), which is what makes adding
//! a node kind cost one honest answer per target, declared here, instead of
//! whatever an `else =>` arm in three serializers happens to do. That is the
//! property `languages/rst/rst.zig` leans on to add kinds only rST has.
//!
//! It is easy to have this property and not notice it is gone. Both halves were
//! broken when the first such kinds (`citation`, `substitution`) arrived: the
//! per-target functions ended in `else => .faithful`, so a new `Kind` inherited
//! `.faithful` in silence; and this module was missing from `root.zig`'s
//! `test {}` block, so Zig — which analyzes lazily — never compiled these
//! switches at all and never ran the probe below. Keep both: the exhaustive
//! spelling, and the import that forces it to be analyzed.
//!
//! ── The table is measured, not asserted ────────────────────────────────────
//! `fidelity` is the single source of truth for what each target can hold, and
//! its entries are not a reading of the serializers — they are what the
//! serializers were OBSERVED to do. `test "the fidelity table matches what the
//! serializers actually do"` builds a minimal document around every kind,
//! serializes it, reparses it with the target's own parser, and asserts the
//! kind is present again exactly when the table claims `.faithful`. A serializer
//! change that alters a round-trip fails here rather than drifting.
//!
//! The probe's default question is "did a node of this kind come back?", which
//! is too coarse twice over, and both refinements exist because the coarse
//! answer was hiding a real loss:
//!
//! * `KindRef.container_named` puts a container's NAME in the match. Djot has
//! only classes to hold a name in, so `:::note` came back as an anonymous
//! div classed `note` — a container, therefore a survival, therefore
//! `.faithful`, for as long as the serializer was deleting the name.
//! * `Probe.intact` asks a second question of the reparsed tree, for a loss
//! that leaves the kind in place. GFM's delimiter row is mandatory, so a
//! header-less table has a header SYNTHESIZED for it: a table goes out and
//! a table comes back, and a row appeared.
//!
//! That mattered immediately: the nearest thing twig already had to a capability
//! model, `syntax.zig`'s `Delims.authorable`, turns out NOT to be one. It is
//! false for both djot's and Markdown's smart-quote containers, but converting
//! to djot round-trips them faithfully (djot's parser makes `double_quoted` out
//! of a bare `"`) while converting to Markdown does not. Same flag, opposite
//! answers — the flag means "an editor gesture may not mint this", which is a
//! different question from "the target can hold this".
//!
//! ── Kind-level and node-level ──────────────────────────────────────────────
//! `fidelity` is a function of `Kind`, so it can answer for anything the parser
//! put IN a node (a container's name) and nothing about what hangs BELOW it.
//! `nodeFidelity` is the second layer, and the one `analyze` calls: it takes
//! the table's answer and downgrades it by what only the node can say. A
//! header-less table is the case that forced it — one `.table` row had to
//! answer for a table that round-trips and a table that came back as a
//! paragraph, and it answered `.faithful` for both.
//!
//! Refinements only ever make an answer WORSE: the table is the ceiling.
//!
//! ── The attribute axis ─────────────────────────────────────────────────────
//! A node's attributes are a side table `Kind` cannot see, and a target can
//! keep the node while losing them: Markdown's serializer writes a `{...}`
//! block for a directive and nothing anywhere else, so a djot paragraph's
//! `{#id .cls}` used to be lost with no warning. `attrsFidelity` is the second
//! measured table — per target, per kind, per KEY CLASS — and `analyze`
//! reports a second `Warning`, `subject == .attrs`, naming the keys, for a
//! node that survives while its attributes do not. Only asked of a node the
//! kind table calls faithful: a node that degrades or drops takes its
//! attributes with it and is reported once, at the node.
//!
//! fig's `Warning.cause` (`format_limitation` vs `explicit_option`) is also
//! absent, because twig has no serializer option that drops anything — every
//! loss here is inherent to the target. It returns when there is a second cause
//! to distinguish.
const std = @import("std");
const Allocator = std.mem.Allocator;
const Writer = std.Io.Writer;
const AST = @import("ast/ast.zig");
const Node = AST.Node;
const format = @import("format.zig");
const Target = format.Target;
/// How much of a node survives a round-trip through a target format: serialize
/// it, hand the result back to that format's own parser, and see what returns.
///
/// The definition is a ROUND-TRIP, which is why this module is indexed by
/// `format.Target` and still needs each target to be readable back. An
/// export-only target (`Target.asFormat() == null`) has no parser to hand the
/// output to, so its fidelity is not measurable this way and the probe below
/// skips it by construction rather than by a hardcoded list. Whoever adds the
/// first such target will find an exhaustive switch in `fidelity` demanding an
/// answer: the honest one is a second axis, not a guess on this one.
pub const Fidelity = enum {
/// The target spells the kind, and reparses that spelling back to it.
faithful,
/// Something is emitted, but the target's parser reads it as a DIFFERENT
/// kind — the content survives, its meaning does not. Markdown has no
/// `^sup^`, so a djot superscript arrives as literal text.
degraded,
/// Nothing is emitted at all: the node and its subtree leave no trace.
dropped,
pub fn isLossy(self: Fidelity) bool {
return self != .faithful;
}
};
/// One thing a conversion to a given format would lose.
pub const Warning = struct {
fidelity: Fidelity,
/// Slash-separated child-index trail from the analyzed root (`"1/0/2"`).
/// Empty means the root itself. Arena-owned by `analyze`.
path: []const u8,
/// The affected node's published kind name — `Kind.kindName`, so a family
/// member reports as itself (`"superscript"`, not `"inline_mark"`).
/// Sentinel-terminated, so the C ABI can hand it out as a `const char *`
/// without copying.
kind: [:0]const u8,
/// What is lost: the node itself, or — the node surviving — only some of
/// its attributes. See `attrsFidelity`.
subject: Subject = .node,
/// For `subject == .attrs`: the keys that do not survive, in the node's own
/// order. Arena-owned by `analyze`. Empty for a `.node` warning.
attrs: []const []const u8 = &.{},
pub const Subject = enum { node, attrs };
/// The default human-readable message, no trailing newline. Bindings may
/// render their own from the structured fields instead.
pub fn render(self: Warning, writer: *Writer, target: Target) Writer.Error!void {
try writer.print("`{s}` at ", .{self.kind});
try writeLoc(writer, self.path);
switch (self.subject) {
.node => switch (self.fidelity) {
.faithful => unreachable, // never recorded
.degraded => try writer.print(" is not spelled by {s} and will reparse as something else", .{@tagName(target)}),
.dropped => try writer.print(" is dropped entirely ({s} cannot write it)", .{@tagName(target)}),
},
.attrs => {
try writer.writeAll(" carries attributes (");
for (self.attrs, 0..) |key, i| {
if (i > 0) try writer.writeAll(", ");
try writer.writeAll(key);
}
switch (self.fidelity) {
.faithful => unreachable, // never recorded
.degraded => try writer.print(") that {s} writes where its parser does not read them back", .{@tagName(target)}),
.dropped => try writer.print(") that {s} cannot write; they are dropped", .{@tagName(target)}),
}
},
}
}
};
fn writeLoc(writer: *Writer, path: []const u8) Writer.Error!void {
if (path.len == 0) {
try writer.writeAll("the document root");
} else {
try writer.writeByte('`');
try writer.writeAll(path);
try writer.writeByte('`');
}
}
pub const AnalyzeError = error{
/// The target has no `serializeFromAst` at all, so there is no conversion to
/// describe — every node would be lost, which is a `format.zig` capability
/// answer (`error.UnsupportedFormat`) rather than a per-node diagnosis.
UnsupportedFormat,
} || Allocator.Error;
/// Walk the subtree at `root_id` as `target`'s serializer would and collect one
/// `Warning` per lossy node. Warnings and their paths are allocated in `arena`.
///
/// A lossy node still has its children walked: a `dropped` container takes its
/// subtree with it, but reporting only the outermost loss would hide that six
/// distinct constructs went missing rather than one.
pub fn analyze(arena: Allocator, ast: *const AST, root_id: Node.Id, target: Target) AnalyzeError![]Warning {
if (format.targetEntryFor(target).serializeFromAst == null) return error.UnsupportedFormat;
var c: Collector = .{ .ast = ast, .arena = arena, .target = target };
try c.walk(root_id, "");
return c.warnings.toOwnedSlice(arena);
}
const Collector = struct {
ast: *const AST,
arena: Allocator,
target: Target,
warnings: std.ArrayList(Warning) = .empty,
fn walk(self: *Collector, id: Node.Id, path: []const u8) AnalyzeError!void {
const kind = self.ast.nodes[id].kind;
const f = nodeFidelity(self.target, self.ast, id);
if (f.isLossy()) {
try self.warnings.append(self.arena, .{
.fidelity = f,
.path = path,
.kind = kind.kindName(),
});
} else {
try self.noteAttrs(id, path);
}
var it = self.ast.children(id);
var i: usize = 0;
while (it.next()) |child| : (i += 1) {
try self.walk(child.id, try childPath(self.arena, path, i));
}
}
/// The attribute half of `walk`, for a node the kind table keeps: one
/// warning naming every key `attrsFidelity` says is lost, at the worse of
/// the two answers when they differ. Nothing for a node with no attributes,
/// and nothing when every key survives.
fn noteAttrs(self: *Collector, id: Node.Id, path: []const u8) AnalyzeError!void {
const attrs = self.ast.attrsOf(id);
if (attrs.isEmpty()) return;
const kind = self.ast.nodes[id].kind;
const table = attrsFidelity(self.target, kind);
var lost: std.ArrayList([]const u8) = .empty;
var f: Fidelity = .faithful;
for (attrs.entries) |kv| {
const kf = table.forKey(kv.key);
if (!kf.isLossy()) continue;
try lost.append(self.arena, try self.arena.dupe(u8, kv.key));
f = worst(f, kf);
}
if (lost.items.len == 0) return;
try self.warnings.append(self.arena, .{
.fidelity = f,
.path = path,
.kind = kind.kindName(),
.subject = .attrs,
.attrs = try lost.toOwnedSlice(self.arena),
});
}
};
fn childPath(arena: Allocator, parent: []const u8, i: usize) Allocator.Error![]const u8 {
if (parent.len == 0) return std.fmt.allocPrint(arena, "{d}", .{i});
return std.fmt.allocPrint(arena, "{s}/{d}", .{ parent, i });
}
// ── the capability table ───────────────────────────────────────────────────
//
// What each target format can hold, per kind. EXHAUSTIVE over `Kind` and over
// the three family enums, which is the property that makes this the gate a new
// kind has to pass: adding one fails this build until it has declared an answer
// for every format, in one place, instead of silently inheriting whatever an
// `else =>` arm in three serializers happens to do.
//
// `str`, and the structural children that only ever appear inside their own
// parent (`list_item`, `row`, `cell`, `term`, …), are `.faithful` everywhere by
// construction: they carry no spelling of their own and ride along with the
// parent that does. A lossy parent is reported once, at the parent.
/// How `kind` survives a conversion to `target`. See the note above the table.
///
/// This is the KIND-level answer, and for most kinds it is the whole answer.
/// Where a loss depends on what hangs BELOW a node rather than on the node
/// itself, `nodeFidelity` is the one to ask — and `analyze` asks it.
pub fn fidelity(target: Target, kind: Node.Kind) Fidelity {
return switch (target) {
// No `serializeFromAst`; `analyze` refuses before reaching the table.
.xml => .dropped,
.djot => djotFidelity(kind),
.markdown => markdownFidelity(kind),
.html => htmlFidelity(kind),
.asciidoc => asciidocFidelity(kind),
};
}
/// `fidelity` for one NODE — the table's answer for its kind, downgraded by
/// whatever only the node's own contents can say.
///
/// A `Kind` carries what the parser put *in* it (a container's name, a
/// heading's level), so a payload-dependent loss is already expressible in the
/// table. What is not is a loss that depends on a node's CHILDREN, and the
/// table has no way to represent one: it is a function of `Kind` alone, so it
/// must answer once for every table in every document.
///
/// GFM is where that bites. Its delimiter row is mandatory, so a table with a
/// header row round-trips and a table without one has a header SYNTHESIZED for
/// it — the same kind, in the same target, with two different answers. The
/// table said `.faithful` for both, and a consumer asking "what will this
/// conversion cost me?" was told nothing about the row that appeared.
///
/// Refinements only ever make the answer WORSE. A node cannot survive better
/// than its kind can, so this can be read as: the table is the ceiling.
pub fn nodeFidelity(target: Target, ast: *const AST, id: Node.Id) Fidelity {
const kind = ast.nodes[id].kind;
const base = fidelity(target, kind);
return switch (kind) {
.table => worst(base, tableFidelity(target, ast, id)),
.container => worst(base, containerFidelity(target, ast, id)),
.section => worst(base, sectionFidelity(target, ast, id)),
.metadata => worst(base, metadataFidelity(target, ast, id)),
else => base,
};
}
/// A metadata block's instance-level answer: whether it is the document's
/// FIRST block. AsciiDoc's only spelling is front matter, which Asciidoctor
/// (and twig's parser) read at the very top of a document and nowhere else;
/// written anywhere else it comes back as a thematic break and text.
fn metadataFidelity(target: Target, ast: *const AST, id: Node.Id) Fidelity {
if (target != .asciidoc) return .faithful;
return if (ast.nodes[ast.root].first_child == id) .faithful else .degraded;
}
/// A section's instance-level answer: whether its title is a LEVEL-ONE one.
///
/// Only AsciiDoc cares. A `= Title` line at the top of a document is the
/// document's title — a header, not a section — so a level-one section
/// written there comes back as the header and its body as the document's.
/// Every deeper section is a section again. `Kind.section` carries no level
/// (the heading child does), so the table cannot see this.
fn sectionFidelity(target: Target, ast: *const AST, id: Node.Id) Fidelity {
if (target != .asciidoc) return .faithful;
const heading = ast.nodes[id].first_child orelse return .faithful;
return switch (ast.nodes[heading].kind) {
.heading => |h| if (h.level == 1) .degraded else .faithful,
else => .faithful,
};
}
/// The more severe of two answers, `faithful` < `degraded` < `dropped`.
fn worst(a: Fidelity, b: Fidelity) Fidelity {
return if (@intFromEnum(b) > @intFromEnum(a)) b else a;
}
/// A table's instance-level answer: whether its FIRST row is a header.
///
/// Only Markdown cares. GFM cannot write a table without a delimiter row and
/// cannot put one anywhere but under the first row, so a header-less table
/// gets an empty header synthesized above it (see the serializer's `.table`
/// arm) — the rows all survive, in order, and the document gains a row it did
/// not have. Djot's separator line is optional and HTML has `<tbody>` without
/// `<thead>`, so for those two a header-less table is simply a table.
fn tableFidelity(target: Target, ast: *const AST, id: Node.Id) Fidelity {
if (target != .markdown) return .faithful;
var rows = ast.tableRows(id);
const first = rows.next() orelse return .faithful;
return if (first.head) .faithful else .degraded;
}
/// A container's instance-level answer: whether it has anything to be spelled
/// WITH.
///
/// Djot's inline container is `[text]{attrs}`, and the attribute block is what
/// makes it a span at all — an anonymous one with no attributes has no
/// spelling, so the wrapper is dropped and only its children come through.
/// `Kind` cannot see this: attributes live in a side table, not in the payload.
fn containerFidelity(target: Target, ast: *const AST, id: Node.Id) Fidelity {
const c = ast.nodes[id].kind.container;
if (target != .djot) return .faithful;
if (c.form != .inline_text or c.name.len > 0) return .faithful;
return if (ast.attrsOf(id).isEmpty()) .dropped else .faithful;
}
/// Djot holds nearly all of twig's vocabulary — unsurprising, since most of it
/// was named after djot's own. The four gaps are real, though, and two of them
/// lose data outright.
fn djotFidelity(kind: Node.Kind) Fidelity {
return switch (kind) {
// Generic markup: djot has no comment, doctype, CDATA or processing
// instruction syntax, and its serializer has no arm for any of them —
// they fall to an inline `else` that renders children, and these have
// none. An HTML comment converted to djot leaves NOTHING behind.
.markup_leaf, .processing_instruction => .dropped,
// A substitution definition renders nothing at the point of definition
// in any format (its body belongs at the use sites), and djot has no
// `|name|` splice to carry it there — so the body goes nowhere. The
// only kind whose CONTENT no target holds; `column` below is dropped by
// all three too, but what it loses is metadata about content that
// survives elsewhere.
.substitution => .dropped,
// A pipe table has no column axis to write one into — djot describes a
// column only through the alignment of its cells. Nothing is emitted,
// so the width and the stub flag go with it. Content is not at risk
// (the cells are children of the ROWS), which is why this is a
// metadata loss rather than a body one, but `Fidelity` measures the
// node and the node does not survive. See `Kind.column`.
.column => .dropped,
// Djot writes a metadata block as a `{lang}`-tagged div, and its own
// parser reads that back as a div, not as metadata.
.metadata => .degraded,
// Djot has no definition-list syntax; the serializer emits the term and
// definition as ordinary blocks.
.definition_list => .degraded,
// No verse construct either: a line block is written as one paragraph
// with hard breaks. The text and the breaks survive, the block does
// not, and each line's `indent` goes with it — djot strips the leading
// space of a continuation line. Unlike `term`/`row`/`cell`, `line` is
// NOT faithful-by-riding-along here, because the parent it rides on
// does not come back either; it is reported on its own.
.line_block, .line => .degraded,
// Djot has ONE footnote registry, so a citation is written as a footnote
// definition and comes back a `footnote`. The content and the
// definition/use link both survive; the second registry does not, which
// is what makes it `degraded` rather than faithful — two labels that
// were distinct across registries can now collide.
.citation => .degraded,
// Djot holds a container's identity in `attrs`, as a class — both the
// fenced div's class line and the bracketed span's `{...}`. So the
// answer turns on the NAME, not on the form: an anonymous container is
// djot's own and returns as itself, while a named one (an rST role, a
// Markdown directive, an HTML tag) comes back anonymous-and-classed.
// The kind survives, its identity does not.
//
// This read `if (c.form == .inline_text) .degraded else .faithful`
// while the serializer was deleting the name outright, and the probe
// agreed, because `want` matched on the TAG — a container came back,
// so a container had survived. `KindRef.container_named` is what makes
// the difference measurable.
.container => |c| if (c.name.len == 0) .faithful else .degraded,
// Math is emitted as `$`…`` — a verbatim wearing a sigil — and djot's
// parser returns the verbatim without it. A citation reference follows
// its definition into the footnote registry; a substitution reference is
// written in its rST spelling and reads as plain text.
.text_leaf => |l| switch (l.kind) {
.inline_math, .display_math, .citation_reference, .substitution_reference => .degraded,
.symb, .verbatim, .url, .email, .footnote_reference => .faithful,
},
.doc,
.para,
.heading,
.thematic_break,
.section,
.code_block,
.raw_block,
.block_quote,
.bullet_list,
.ordered_list,
.task_list,
.table,
.list_item,
.task_list_item,
.definition_list_item,
.term,
.definition,
.row,
.cell,
.caption,
.footnote,
.reference,
.str,
.soft_break,
.hard_break,
.non_breaking_space,
.raw_inline,
.smart_punctuation,
.link,
.image,
.inline_mark,
=> .faithful,
};
}
/// Markdown spells strictly less than djot — three inline marks against nine,
/// no generic containers, no smart punctuation of its own — so this is where
/// most conversion loss actually happens.
fn markdownFidelity(kind: Node.Kind) Fidelity {
return switch (kind) {
// CommonMark has no heading-scoped section; the children are emitted
// flat and the wrapper is gone.
.section => .degraded,
// A raw block is written as a fenced code block, which reparses as one.
.raw_block => .degraded,
// Markdown has no `:::name` fenced div in CommonMark; the serializer's
// directive spelling is an extension its own parser does not read back
// without one.
.container => .degraded,
// Written as raw HTML text, which comes back as a raw block/inline or
// as escaped characters — never as the leaf it was.
.markup_leaf, .processing_instruction, .raw_inline => .degraded,
// Emitted as its ASCII spelling (`...`, `"`), which CommonMark reads as
// ordinary characters.
.smart_punctuation => .degraded,
// ` ` — an entity, so it returns as text.
.non_breaking_space => .degraded,
// Only `strong`, `emph` and GFM's `delete` survive. The rest carry the
// extension spellings `markdown/syntax.zig` records for exactly this
// conversion (`==mark==`, `^sup^`, `{+ins+}`), which CommonMark reads as
// literal text — better than dropping the node, but not the same node.
// The two smart-quote containers are the case that proves `authorable`
// is not this question: they are `authorable = false` in BOTH tables,
// yet faithful in djot and degraded here.
.inline_mark => |m| switch (m) {
.strong, .emph, .delete => .faithful,
.mark, .superscript, .subscript, .insert, .double_quoted, .single_quoted => .degraded,
},
// Both new definitions behave exactly as they do for djot, and for the
// same reasons — Markdown's footnote extension is djot's registry over
// again, and it has no substitution mechanism either.
.citation => .degraded,
.substitution => .dropped,
// Same spelling and same loss as djot's — see that arm.
.line_block, .line => .degraded,
// GFM's pipe table has no column axis either — same as djot.
.column => .dropped,
.text_leaf => |l| switch (l.kind) {
.verbatim, .url, .email, .footnote_reference => .faithful,
.symb, .inline_math, .display_math, .citation_reference, .substitution_reference => .degraded,
},
.doc,
.para,
.heading,
.thematic_break,
.code_block,
.metadata,
.block_quote,
.bullet_list,
.ordered_list,
.task_list,
.definition_list,
.table,
.list_item,
.task_list_item,
.definition_list_item,
.term,
.definition,
.row,
.cell,
.caption,
.footnote,
.reference,
.str,
.soft_break,
.hard_break,
.link,
.image,
=> .faithful,
};
}
/// AsciiDoc holds nearly the whole vocabulary — it has a native spelling for
/// most of what djot and Markdown between them invented — and what it loses
/// it loses to a reparse that reads the spelling as a near neighbour.
fn asciidocFidelity(kind: Node.Kind) Fidelity {
return switch (kind) {
// Generic markup: a comment is written as a comment BLOCK, which
// produces no node at all; a doctype and a processing instruction
// have no spelling and are not written; a CDATA section's text is
// kept as text.
.markup_leaf => |l| switch (l.kind) {
.comment, .doctype => .dropped,
.cdata => .degraded,
},
.processing_instruction => .dropped,
// A substitution definition is a body attribute entry (`:name:
// value`), which reparses as the same definition, and its use as an
// attribute reference — the one target where both survive.
.substitution => .faithful,
// No column axis in an AsciiDoc table either (`cols` describes
// widths, not a node); nothing is emitted for a `column`.
.column => .dropped,
// Front matter is written as YAML front matter, which the parser
// reads back only at the very start of a document — `nodeFidelity`
// downgrades a `metadata` block that sits anywhere else.
.metadata => .faithful,
// A citation is an anchor over its body plus `<<xref>>`s at its
// uses: both halves survive, the citation registry does not.
.citation => .degraded,
// Written as a delimited block whose style is the container's name:
// an admonition, example, sidebar or open block returns as itself; any
// other name (a Markdown `:::box`, an HTML tag, an inline `:abbr`)
// comes back as an open block or a styled span classed with the name
// — the kind survives, the name becomes a role. An ANONYMOUS block
// container is the open block itself, and the parser names that one
// `open`: the node returns, its namelessness does not.
.container => |c| if (isNativeContainer(c.name)) .faithful else .degraded,
.inline_mark => |m| switch (m) {
.strong, .emph, .mark, .superscript, .subscript, .double_quoted, .single_quoted => .faithful,
// Asciidoctor's convention is a role its stylesheet knows; the
// reparse reads a styled span, not a mark.
.insert, .delete => .degraded,
},
.text_leaf => |l| switch (l.kind) {
.verbatim, .url, .email, .footnote_reference, .substitution_reference => .faithful,
// Math is written as `stem:[…]`, and comes back as a `stem` macro
// whose display/inline distinction is gone: the inline form
// survives as itself, the display form arrives as the inline one.
.inline_math => .faithful,
.display_math => .degraded,
// No shortcodes: `:name:` is text.
.symb => .degraded,
// A citation reference is written as a cross reference and reads
// back as a link to the citation's anchor.
.citation_reference => .degraded,
},
// A link-reference definition is resolved into its uses and not
// written; a footnote definition is written at its reference, and
// the reparse re-creates one from it.
.reference => .degraded,
.footnote => .faithful,
// A heading is a section title, and every section title opens a
// section: a bare `heading` (one not already in a section) comes
// back wrapped in one. The kind survives; the tree gains a section.
.heading => .faithful,
.doc,
.para,
.thematic_break,
.section,
.code_block,
.raw_block,
.block_quote,
.bullet_list,
.ordered_list,
.task_list,
.definition_list,
.line_block,
.table,
.list_item,
.task_list_item,
.definition_list_item,
.term,
.definition,
.line,
.row,
.cell,
.caption,
.str,
.soft_break,
.hard_break,
.non_breaking_space,
.raw_inline,
.smart_punctuation,
.link,
.image,
=> .faithful,
};
}
/// The container names AsciiDoc spells as their own block: the admonitions,
/// and the three delimited parents.
fn isNativeContainer(name: []const u8) bool {
inline for (.{ "note", "tip", "important", "warning", "caution", "example", "sidebar", "open", "page-break", "kbd", "abstract", "partintro" }) |n| {
if (std.mem.eql(u8, name, n)) return true;
}
return false;
}
/// HTML is a RENDER target, not a source syntax, and the asymmetry shows: it
/// spells the generic-markup kinds djot and Markdown cannot, and loses the
/// lightweight-markup semantics they keep. Everything here is measured against
/// re-parsing the rendered HTML, which is what a caller converting to HTML and
/// reading it back actually gets.
fn htmlFidelity(kind: Node.Kind) Fidelity {
return switch (kind) {
// The generic-markup kinds are HTML's own, and survive as themselves.
.processing_instruction => .faithful,
.markup_leaf => |l| switch (l.kind) {
.comment, .doctype => .faithful,
// Rendered escaped rather than as a `<![CDATA[…]]>` section.
.cdata => .degraded,
},
// A raw block's payload is written through verbatim, so the wrapper is
// gone and what remains is whatever those bytes parse as.
.raw_block, .raw_inline => .degraded,
// Rendered as its config text inside a comment or dropped from the body.
.metadata => .degraded,
// Checkbox inputs inside an ordinary list: the list survives, the
// task-ness does not.
.task_list, .task_list_item => .degraded,
// Definition lists render as `<dl>`, which twig's HTML parser has no
// semantic mapping for yet.
.definition_list, .definition_list_item, .term, .definition => .degraded,
// The one target that spells a line block the way its source format
// does — docutils' `<div class="line-block">`/`<div class="line">`,
// indent and all (see the serializer's `renderLineBlock`). It still
// degrades, for the `<dl>` reason: a classed div reads back as a
// generic `container`, so the output is right and the KIND is gone.
.line_block, .line => .degraded,
// Footnote and link-reference DEFINITIONS are resolved at render time —
// the reference is inlined into an `<a href>`/endnote and the definition
// node itself has no output of its own.
.footnote, .reference => .degraded,
// A citation is the one definition this printer writes IN PLACE, as
// docutils' HTML5 writer does, so it does leave output — a
// `<div class="citation">` that twig's HTML parser reads back as a
// generic `container`.
.citation => .degraded,
// Nothing is written, here or anywhere: the printer does not resolve a
// substitution (no side table exists to resolve it against until the rST
// parser lands), and a definition has no output of its own. The body is
// lost outright, which is the strongest reason this whole module was
// built before the vocabulary that needs it.
.substitution => .dropped,
// The one target that HAS the construct — `<colgroup><col>` — and still
// does not write it. Emitting a bare `<col>` among the rows would be
// invalid (HTML's table content model puts `col` inside a `colgroup`),
// so spelling this means teaching the serializer's thead/tbody state
// machine a third group, and reading it back means mapping `<col>` in
// the HTML parser. Both are worth doing and neither is done, so the
// honest entry today is the same as the other two targets'. This is the
// arm to revisit first if `Kind.column` ever grows a typed payload.
.column => .dropped,
// A NAMED container is HTML's own — `<video>` goes out as `<video>` and
// comes back as one. An ANONYMOUS one cannot be: there is no such thing
// as an unnamed HTML tag, so the serializer picks `div` or `span` by
// level and the parser reads that back as a container NAMED "div" or
// "span". The node survives; the fact that it had no name does not, and
// afterwards it is indistinguishable from a document that really did
// write `<div>`.
.container => |c| if (c.name.len == 0) .degraded else .faithful,
// Rendered as Unicode glyphs, which come back as ordinary text.
.smart_punctuation, .non_breaking_space => .degraded,
.inline_mark => |m| switch (m) {
// `<q>` is not what twig's HTML parser makes a `double_quoted` from.
.double_quoted, .single_quoted => .degraded,
.strong, .emph, .mark, .superscript, .subscript, .insert, .delete => .faithful,
},
.text_leaf => |l| switch (l.kind) {
.verbatim => .faithful,
// An autolink renders as `<a href>` and returns as a `link`; math
// renders as a classed span; a footnote reference becomes a
// numbered `<sup>` whose label is gone; a citation reference is an
// `<a href="#label">` that likewise returns as a `link`; an
// unresolved substitution reference is written as its own `|name|`
// spelling and returns as text.
.url, .email, .inline_math, .display_math, .symb, .footnote_reference, .citation_reference, .substitution_reference => .degraded,
},
.doc,
.para,
.heading,
.thematic_break,
.section,
.code_block,
.block_quote,
.bullet_list,
.ordered_list,
.table,
.list_item,
.row,
.cell,
.caption,
.str,
.soft_break,
.hard_break,
.link,
.image,
=> .faithful,
};
}
// ── the attribute axis ─────────────────────────────────────────────────────
//
// What each target does with a node's ATTRIBUTES — the side table `Kind`
// cannot see, and the axis the module header used to name as "not covered
// yet". Measured the way the kind table is: `test "the attribute table matches
// what the serializers actually do"` puts four attributes on every probed node,
// round-trips it through each target, and asks PER KEY whether the value came
// back on the node (`faithful`), was written somewhere the target's own parser
// does not read as that node's attribute (`degraded` — djot and AsciiDoc both
// read a heading's attribute line back as the SECTION's), or was never
// written (`dropped`).
//
// The answer is per KEY CLASS, because the formats are ragged at exactly that
// grain: AsciiDoc's inline `[#id.role]` keeps an id and a class on a mark and
// has nowhere for a `data-k`; its link macro keeps a role and drops the id.
// Three classes — `id`, `class`, and everything else — are what the spellings
// distinguish, plus `title`, the one key a format spells from the side table
// in a position of its own (Markdown's `[label]: dest "title"`). The probe's two
// "other" keys, `data-k` and `title`, are there to catch a fifth.
//
// Only asked of a node the KIND table calls faithful. A node that degrades or
// drops takes its attributes with it and is reported once, at the node — so
// each per-target function below answers for the kinds that survive there, and
// answers `.dropped` for the rest, where the answer is never consulted. The
// switches are exhaustive anyway, for the reason the kind table's are.
/// One target's answer about one kind's attributes, per key class.
pub const AttrsFidelity = struct {
id: Fidelity,
class: Fidelity,
other: Fidelity,
/// The `title` key where it differs from `other`; `null` means it does not.
title: ?Fidelity = null,
pub fn all(f: Fidelity) AttrsFidelity {
return .{ .id = f, .class = f, .other = f };
}
pub fn forKey(self: AttrsFidelity, key: []const u8) Fidelity {
if (std.mem.eql(u8, key, "id")) return self.id;
if (std.mem.eql(u8, key, "class")) return self.class;
if (std.mem.eql(u8, key, "title")) return self.title orelse self.other;
return self.other;
}
};
/// How `kind`'s attributes survive a conversion to `target`. See the note
/// above; the kind-level `fidelity` is the ceiling, and this is consulted only
/// under it.
pub fn attrsFidelity(target: Target, kind: Node.Kind) AttrsFidelity {
return switch (target) {
.xml => .all(.dropped),
.djot => djotAttrsFidelity(kind),
.markdown => markdownAttrsFidelity(kind),
.html => htmlAttrsFidelity(kind),
.asciidoc => asciidocAttrsFidelity(kind),
};
}
/// Djot can spell an attribute block on any block or inline, and its
/// serializer writes one in exactly five places.
fn djotAttrsFidelity(kind: Node.Kind) AttrsFidelity {
return switch (kind) {
// The line before a paragraph, a reference definition or a section's
// heading; the line before a fenced div, or after a bracketed span.
// Read back in full.
.para, .container, .reference, .section => .all(.faithful),
// Written as the line before the heading, which is djot's block
// spelling — and, as in AsciiDoc, a block that names an id is the
// SECTION's when djot reads it back (the parser moves the whole set,
// after djot.js), so the heading node comes back bare. Without an id
// the set stays on the heading; the probe carries one.
.heading => .all(.degraded),
// Every other kind carries its attributes into the serializer and out
// the far side of nothing: no `{…}` is written for a quote, a list, a
// fence, a table, a link, an image or a mark, though djot could hold
// one on each.
.doc,
.thematic_break,
.code_block,
.raw_block,
.metadata,
.block_quote,
.bullet_list,
.ordered_list,
.task_list,
.definition_list,
.line_block,
.table,
.list_item,
.task_list_item,
.definition_list_item,
.term,
.definition,
.line,
.row,
.cell,
.column,
.caption,
.footnote,
.citation,
.substitution,
.str,
.soft_break,
.hard_break,
.non_breaking_space,
.text_leaf,
.raw_inline,
.smart_punctuation,
.link,
.image,
.inline_mark,
.markup_leaf,
.processing_instruction,
=> .all(.dropped),
};
}
/// Markdown has no attribute spelling of its own. A block's attributes are
/// written on a `<div>` around it — raw HTML, which every reader passes
/// through and the DEFAULT parser reads back as two raw blocks beside the
/// block, so the answer here is `degraded`; under `ParseOptions.html_elements`
/// the same bytes pair into a container whose sole child is the block, which
/// is as close as the format comes. An inline's have nowhere to go at all:
/// no `*em*` spelling takes one.
fn markdownAttrsFidelity(kind: Node.Kind) AttrsFidelity {
return switch (kind) {
.para,
.heading,
.thematic_break,
.section,
.code_block,
.raw_block,
.block_quote,
.bullet_list,
.ordered_list,
.task_list,
.definition_list,
.line_block,
.table,
=> .all(.degraded),
// A reference definition's title is part of its spelling, `[label]:
// dest "title"`, and comes back; nothing else on it is written.
.reference => .{ .id = .dropped, .class = .dropped, .other = .dropped, .title = .faithful },
// Front matter is never wrapped — it has to stay the first block.
.metadata,
.doc,
.list_item,
.task_list_item,
.definition_list_item,
.term,
.definition,
.line,
.row,
.cell,
.column,
.caption,
.footnote,
.citation,
.substitution,
.str,
.soft_break,
.hard_break,
.non_breaking_space,
.text_leaf,
.raw_inline,
.smart_punctuation,
.link,
.image,
.inline_mark,
.container,
.markup_leaf,
.processing_instruction,
=> .all(.dropped),
};
}
/// HTML writes every attribute into the tag it renders, so a kind with an
/// element keeps them all and a kind without one has nowhere to put them.
fn htmlAttrsFidelity(kind: Node.Kind) AttrsFidelity {
return switch (kind) {
.para,
.heading,
.thematic_break,
.section,
.code_block,
.block_quote,
.bullet_list,
.ordered_list,
.table,
.list_item,
.row,
.cell,
.caption,
.link,
.image,
.inline_mark,
.container,
.text_leaf,
=> .all(.faithful),
// Rendered as a bare `<br>`, whatever it carries.
.hard_break => .all(.dropped),
// No element of their own: plain text, a comment, a doctype, a
// processing instruction — and the kinds HTML does not keep anyway.
.doc,
.raw_block,
.metadata,
.task_list,
.definition_list,
.line_block,
.task_list_item,
.definition_list_item,
.term,
.definition,
.line,
.column,
.footnote,
.reference,
.citation,
.substitution,
.str,
.soft_break,
.non_breaking_space,
.raw_inline,
.smart_punctuation,
.markup_leaf,
.processing_instruction,
=> .all(.dropped),
};
}
/// AsciiDoc's block attribute line — `[#id.role,key=value]` above the block
/// — carries everything and is read back in full; its inline spellings carry
/// less, and one block's line is read as another's.
fn asciidocAttrsFidelity(kind: Node.Kind) AttrsFidelity {
return switch (kind) {
.para,
.thematic_break,
.code_block,
.raw_block,
.block_quote,
.bullet_list,
.ordered_list,
.task_list,
.definition_list,
.line_block,
.container,
=> .all(.faithful),
// The same line and the same `.Title` line — which the parser reads
// back as the table's CAPTION, the one block whose title is a child
// and not an attribute.
.table => .{ .id = .faithful, .class = .faithful, .other = .faithful, .title = .degraded },
// A section title's attribute line is the SECTION's: Asciidoctor and
// twig both read `[#id]` above `== Title` as the section's id, and
// the heading node comes back bare.
.heading => .all(.degraded),
// The block image macro's `[alt,id=…]` list comes back on the
// paragraph the macro parses into.
.image => .all(.degraded),
// `[#id.role]` before a formatting pair keeps an id and a role; the
// grammar has no slot for any other key. The same prefix before a
// `#mark#` or a quoted pair is written and read as a styled span (or
// as text), so there the id and role are written and lost.
.inline_mark => |m| switch (m) {
.emph, .strong, .superscript, .subscript => .{ .id = .faithful, .class = .faithful, .other = .dropped },
.mark, .double_quoted, .single_quoted => .{ .id = .degraded, .class = .degraded, .other = .dropped },
.insert, .delete => .all(.dropped),
},
.text_leaf => |l| switch (l.kind) {
.verbatim => .{ .id = .faithful, .class = .faithful, .other = .dropped },
.symb, .inline_math, .display_math, .url, .email, .footnote_reference, .citation_reference, .substitution_reference => .all(.dropped),
},
// The link macro's attribute list keeps a role and nothing else.
.link => .{ .id = .dropped, .class = .faithful, .other = .dropped },
.doc,
.section,
.metadata,
.list_item,
.task_list_item,
.definition_list_item,
.term,
.definition,
.line,
.row,
.cell,
.column,
.caption,
.footnote,
.reference,
.citation,
.substitution,
.str,
.soft_break,
.hard_break,
.non_breaking_space,
.raw_inline,
.smart_punctuation,
.markup_leaf,
.processing_instruction,
=> .all(.dropped),
};
}
// ── tests ──────────────────────────────────────────────────────────────────
const testing = std.testing;
const select = @import("ast/select.zig");
/// The targets a round-trip claim can be made about at all: those Twig can
/// WRITE (`serializeFromAst`) and can also READ BACK (`reads_back_as`). Derived
/// from the `targets` table rather than listed, so a new row is either measured
/// automatically or excluded for a reason the table itself states — an
/// export-only target drops out here by construction, and no probe is ever
/// silently skipped by a hardcoded list going stale.
const round_trippable: []const Target = blk: {
var out: []const Target = &.{};
for (format.targets) |e| {
if (e.serializeFromAst != null and e.reads_back_as != null) out = out ++ [_]Target{e.id};
}
break :blk out;
};
test "the probe covers every target that can be measured, and today that is four" {
try testing.expectEqual(@as(usize, 4), round_trippable.len);
try testing.expectEqualSlices(Target, &.{ .djot, .markdown, .html, .asciidoc }, round_trippable);
}
test "analyze refuses a target with no serializer rather than reporting every node" {
var b = AST.Builder.init(testing.allocator);
defer b.deinit();
const root = try b.addContainer(.doc, &.{});
var ast = try b.finish(root);
defer ast.deinit();
var arena = std.heap.ArenaAllocator.init(testing.allocator);
defer arena.deinit();
try testing.expectError(error.UnsupportedFormat, analyze(arena.allocator(), &ast, ast.root, .xml));
}
test "two tables, same kind, different answers — and only the header-less one is reported" {
const allocator = testing.allocator;
var b = AST.Builder.init(allocator);
defer b.deinit();
// [0] a table WITH a header row: GFM's delimiter row has something to
// describe, and the conversion costs nothing.
const h = try b.addContainer(.{ .cell = .{ .head = true, .alignment = .default } }, &.{try str(&b, "H")});
const hr = try b.addContainer(.{ .row = .{ .head = true } }, &.{h});
const with = try b.addContainer(.table, &.{hr});
// [1] the same table WITHOUT one, which gets a header synthesized above it.
const c = try b.addContainer(.{ .cell = .{ .head = false, .alignment = .default } }, &.{try str(&b, "a")});
const r = try b.addContainer(.{ .row = .{ .head = false } }, &.{c});
const without = try b.addContainer(.table, &.{r});
var ast = try b.finish(try b.addContainer(.doc, &.{ with, without }));
defer ast.deinit();
var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit();
const warnings = try analyze(arena.allocator(), &ast, ast.root, .markdown);
// One warning, on the second table. The kind-level table cannot tell these
// apart — this is the whole reason `nodeFidelity` exists.
try testing.expectEqual(@as(usize, 1), warnings.len);
try testing.expectEqualStrings("table", warnings[0].kind);
try testing.expectEqualStrings("1", warnings[0].path);
try testing.expectEqual(Fidelity.degraded, warnings[0].fidelity);
// Djot's separator line is optional, so neither table costs anything there
// — the answer is a property of the (document, target) PAIR, not of either
// one alone.
const djot_warnings = try analyze(arena.allocator(), &ast, ast.root, .djot);
try testing.expectEqual(@as(usize, 0), djot_warnings.len);
}
test "an HTML comment converted to djot is reported as dropped, and really is" {
const allocator = testing.allocator;
var b = AST.Builder.init(allocator);
defer b.deinit();
const c = try b.addLeaf(.{ .markup_leaf = .{ .kind = .comment, .text = " secret " } });
const p = try b.addContainer(.para, &.{try b.addLeaf(.{ .str = "hi" })});
const root = try b.addContainer(.doc, &.{ p, c });
var ast = try b.finish(root);
defer ast.deinit();
var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit();
const warnings = try analyze(arena.allocator(), &ast, ast.root, .djot);
try testing.expectEqual(@as(usize, 1), warnings.len);
try testing.expectEqual(Fidelity.dropped, warnings[0].fidelity);
try testing.expectEqualStrings("comment", warnings[0].kind);
try testing.expectEqualStrings("1", warnings[0].path);
// And the warning is true: the comment leaves no trace in the djot output.
const src = try format.targetEntryFor(.djot).serializeFromAst.?(allocator, &ast);
defer allocator.free(src);
try testing.expect(std.mem.indexOf(u8, src, "secret") == null);
}
test "a substitution's body is dropped by every target but AsciiDoc, and is reported as such" {
const allocator = testing.allocator;
var b = AST.Builder.init(allocator);
defer b.deinit();
const def = try b.addContainer(.{ .substitution = .{ .label = "RST" } }, &.{try b.addLeaf(.{ .str = "reStructuredText" })});
const ref = try b.addLeaf(.{ .text_leaf = .{ .kind = .substitution_reference, .text = "RST" } });
const body = try b.addContainer(.para, &.{ref});
const root = try b.addContainer(.doc, &.{ body, def });
var ast = try b.finish(root);
defer ast.deinit();
for (round_trippable) |target| {
var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit();
const warnings = try analyze(arena.allocator(), &ast, ast.root, target);
if (target == .asciidoc) {
// The one target with the construct: a body attribute entry and
// an attribute reference, both of which read back as themselves.
try testing.expectEqual(@as(usize, 0), warnings.len);
const src = try format.targetEntryFor(target).serializeFromAst.?(allocator, &ast);
defer allocator.free(src);
try testing.expect(std.mem.indexOf(u8, src, ":RST: reStructuredText") != null);
continue;
}
// The definition is `dropped` and its use `degraded` — two warnings,
// never zero, for every other target twig can write.
try testing.expectEqual(@as(usize, 2), warnings.len);
try testing.expectEqual(Fidelity.degraded, warnings[0].fidelity);
try testing.expectEqualStrings("substitution_reference", warnings[0].kind);
try testing.expectEqual(Fidelity.dropped, warnings[1].fidelity);
try testing.expectEqualStrings("substitution", warnings[1].kind);
// And `dropped` is the literal truth: the body reaches no output.
const src = try format.targetEntryFor(target).serializeFromAst.?(allocator, &ast);
defer allocator.free(src);
try testing.expect(std.mem.indexOf(u8, src, "reStructuredText") == null);
}
}
test "a citation survives a conversion as a footnote, definition and use together" {
const allocator = testing.allocator;
var b = AST.Builder.init(allocator);
defer b.deinit();
const p = try b.addContainer(.para, &.{try b.addLeaf(.{ .str = "Deep Thought." })});
const def = try b.addContainer(.{ .citation = .{ .label = "CIT2002" } }, &.{p});
const ref = try b.addLeaf(.{ .text_leaf = .{ .kind = .citation_reference, .text = "CIT2002" } });
const body = try b.addContainer(.para, &.{ try b.addLeaf(.{ .str = "see " }), ref });
const root = try b.addContainer(.doc, &.{ body, def });
var ast = try b.finish(root);
defer ast.deinit();
// Flattened into djot's one footnote registry: `degraded`, not `dropped` —
// both halves reach the output and still name each other, which is what
// separates this from the substitution case above.
const src = try format.targetEntryFor(.djot).serializeFromAst.?(allocator, &ast);
defer allocator.free(src);
try testing.expect(std.mem.indexOf(u8, src, "[^CIT2002]") != null);
try testing.expect(std.mem.indexOf(u8, src, "[^CIT2002]: Deep Thought.") != null);
}
test "a warning renders with its path and target" {
var buf: [256]u8 = undefined;
var w = Writer.fixed(&buf);
const warning: Warning = .{ .fidelity = .degraded, .path = "0/1", .kind = "superscript" };
try warning.render(&w, .markdown);
try testing.expectEqualStrings(
"`superscript` at `0/1` is not spelled by markdown and will reparse as something else",
w.buffered(),
);
}
// ── the probe that keeps the table honest ──────────────────────────────────
//
// For every kind, build a minimal document containing it, serialize it to each
// target, reparse with that target's own parser, and assert the kind comes back
// exactly when `fidelity` says `.faithful`. This is what makes the table a
// measurement instead of a claim.
//
// Only the `.faithful` bit is asserted. Separating `degraded` from `dropped`
// needs to know whether ANY trace of the node reached the output, and for the
// payload-less kinds (`smart_punctuation`, `non_breaking_space`) there is no
// content to look for — so that distinction is documented per entry from the
// serializers rather than probed, and is the weaker half of this table.
fn str(b: *AST.Builder, s: []const u8) !Node.Id {
return b.addLeaf(.{ .str = s });
}
fn blockDoc(b: *AST.Builder, kind: Node.Kind, children: []const Node.Id) !Node.Id {
return b.addContainer(.doc, &.{try b.addContainer(kind, children)});
}
fn inlineDoc(b: *AST.Builder, kind: Node.Kind, children: []const Node.Id) !Node.Id {
const n = try b.addContainer(kind, children);
return b.addContainer(.doc, &.{try b.addContainer(.para, &.{n})});
}
fn buildLineBlock(b: *AST.Builder) anyerror!Node.Id {
const l1 = try b.addContainer(.{ .line = .{} }, &.{try str(b, "one")});
const l2 = try b.addContainer(.{ .line = .{ .indent = 1 } }, &.{try str(b, "two")});
return blockDoc(b, .line_block, &.{ l1, l2 });
}
const Probe = struct {
label: []const u8,
/// What the built document is being probed FOR, and the kind whose
/// `fidelity` entry the round-trip must agree with.
want: AST.KindRef,
kind: Node.Kind,
build: *const fn (*AST.Builder) anyerror!Node.Id,
/// An extra condition on the REPARSED tree, for a loss that leaves the
/// kind in place. `want` asks whether the node came back; this asks
/// whether it came back UNCHANGED, and only a probe with something to
/// check sets it.
///
/// The header-less table is why it exists. GFM's mandatory delimiter row
/// means the serializer synthesizes a header the document did not have, so
/// a `table` goes out and a `table` comes back — `want` is satisfied, and
/// the round-trip still added a row. Without somewhere to say that, the
/// only way to keep the probe passing would have been to call the
/// synthesized header faithful.
intact: ?*const fn (*const AST) bool = null,
};
/// Only ever called with a `round_trippable` target, so both unwraps below hold:
/// the write half comes from the TARGET row and the read half from the INPUT row
/// the target names, which is the whole shape of the two-table split.
fn survives(
allocator: Allocator,
ast: *const AST,
target: Target,
want: AST.KindRef,
intact: ?*const fn (*const AST) bool,
) !bool {
const src = try format.targetEntryFor(target).serializeFromAst.?(allocator, ast);
defer allocator.free(src);
const cfg = format.ParseConfig{};
var back = try format.entryFor(target.asFormat().?).parseToAst(&cfg, allocator, src);
defer back.deinit();
var found = false;
for (back.ast.nodes) |n| {
if (want.matches(n.kind)) {
found = true;
break;
}
}
if (!found) return false;
const check = intact orelse return true;
return check(&back.ast);
}
/// The id of the node a probe is ABOUT, within the document it built — the
/// first whose kind tag matches `p.kind`'s. Probes build a minimal document
/// around exactly one node of the kind under test, so "first match" is that
/// node. Needed because the claim being checked is now `nodeFidelity`'s, which
/// is a question about a node rather than about a kind.
fn probedNode(ast: *const AST, kind: Node.Kind) ?Node.Id {
for (ast.nodes, 0..) |n, i| {
if (std.meta.activeTag(n.kind) == std.meta.activeTag(kind)) return @intCast(i);
}
return null;
}
test "the fidelity table matches what the serializers actually do" {
const allocator = testing.allocator;
for (probes) |p| {
var b = AST.Builder.init(allocator);
defer b.deinit();
const root = try p.build(&b);
var ast = try b.finish(root);
defer ast.deinit();
for (round_trippable) |target| {
const claimed = nodeFidelity(target, &ast, probedNode(&ast, p.kind).?);
const observed = try survives(allocator, &ast, target, p.want, p.intact);
if ((claimed == .faithful) != observed) {
std.debug.print(
"\nfidelity({s}, {s}) claims .{s}, but the kind {s} a round-trip\n",
.{ @tagName(target), p.label, @tagName(claimed), if (observed) "SURVIVES" else "does NOT survive" },
);
return error.FidelityTableStale;
}
}
}
}
const probes = [_]Probe{
.{ .label = "heading", .want = .{ .tag = .heading }, .kind = .{ .heading = .{ .level = 2 } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return blockDoc(b, .{ .heading = .{ .level = 2 } }, &.{try str(b, "x")});
}
}.f },
.{ .label = "section", .want = .{ .tag = .section }, .kind = .section, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const h = try b.addContainer(.{ .heading = .{ .level = 1 } }, &.{try str(b, "t")});
const p = try b.addContainer(.para, &.{try str(b, "x")});
return blockDoc(b, .section, &.{ h, p });
}
}.f },
.{ .label = "block_quote", .want = .{ .tag = .block_quote }, .kind = .block_quote, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const p = try b.addContainer(.para, &.{try str(b, "x")});
return blockDoc(b, .block_quote, &.{p});
}
}.f },
.{ .label = "thematic_break", .want = .{ .tag = .thematic_break }, .kind = .thematic_break, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return blockDoc(b, .thematic_break, &.{});
}
}.f },
.{ .label = "code_block", .want = .{ .tag = .code_block }, .kind = .{ .code_block = .{ .lang = null, .text = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return blockDoc(b, .{ .code_block = .{ .lang = "zig", .text = "x\n" } }, &.{});
}
}.f },
.{ .label = "raw_block", .want = .{ .tag = .raw_block }, .kind = .{ .raw_block = .{ .format = "", .text = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return blockDoc(b, .{ .raw_block = .{ .format = "html", .text = "<x>\n" } }, &.{});
}
}.f },
.{ .label = "metadata", .want = .{ .tag = .metadata }, .kind = .{ .metadata = .{ .lang = "", .text = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return blockDoc(b, .{ .metadata = .{ .lang = "yaml", .text = "a: 1\n" } }, &.{});
}
}.f },
.{ .label = "bullet_list", .want = .{ .tag = .bullet_list }, .kind = .{ .bullet_list = .{ .tight = true } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const p = try b.addContainer(.para, &.{try str(b, "x")});
const li = try b.addContainer(.list_item, &.{p});
return blockDoc(b, .{ .bullet_list = .{ .tight = true } }, &.{li});
}
}.f },
.{ .label = "ordered_list", .want = .{ .tag = .ordered_list }, .kind = .{ .ordered_list = .{ .numbering = .decimal, .tight = true, .start = 1 } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const p = try b.addContainer(.para, &.{try str(b, "x")});
const li = try b.addContainer(.list_item, &.{p});
return blockDoc(b, .{ .ordered_list = .{ .numbering = .decimal, .tight = true, .start = 1 } }, &.{li});
}
}.f },
.{ .label = "task_list", .want = .{ .tag = .task_list }, .kind = .{ .task_list = .{ .tight = true } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const p = try b.addContainer(.para, &.{try str(b, "x")});
const li = try b.addContainer(.{ .task_list_item = .{ .checked = true } }, &.{p});
return blockDoc(b, .{ .task_list = .{ .tight = true } }, &.{li});
}
}.f },
.{ .label = "definition_list", .want = .{ .tag = .definition_list }, .kind = .definition_list, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const t = try b.addContainer(.term, &.{try str(b, "t")});
const p = try b.addContainer(.para, &.{try str(b, "d")});
const d = try b.addContainer(.definition, &.{p});
const item = try b.addContainer(.definition_list_item, &.{ t, d });
return blockDoc(b, .definition_list, &.{item});
}
}.f },
// A two-line verse with an indented second line, probed for the BLOCK and
// for the LINE separately — the two claims are independent, and a probe
// asserting both would be satisfied by either surviving alone.
.{ .label = "line_block", .want = .{ .tag = .line_block }, .kind = .line_block, .build = buildLineBlock },
.{ .label = "line", .want = .{ .tag = .line }, .kind = .{ .line = .{} }, .build = buildLineBlock },
.{ .label = "table", .want = .{ .tag = .table }, .kind = .table, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const cap = try b.addContainer(.caption, &.{});
const c1 = try b.addContainer(.{ .cell = .{ .head = true, .alignment = .default } }, &.{try str(b, "a")});
const r1 = try b.addContainer(.{ .row = .{ .head = true } }, &.{c1});
const c2 = try b.addContainer(.{ .cell = .{ .head = false, .alignment = .default } }, &.{try str(b, "1")});
const r2 = try b.addContainer(.{ .row = .{ .head = false } }, &.{c2});
return blockDoc(b, .table, &.{ cap, r1, r2 });
}
}.f },
// The same table with NO header row — the instance case, and the one that
// sent a downstream consumer hand-rolling a `pipes_that_are_not_a_table`
// heuristic because the table above answered for both. `nodeFidelity` is
// what lets these two rows disagree; before it, one `.table => .faithful`
// had to cover a table that round-trips and a table that came back as a
// paragraph.
.{
.label = "table(header-less)",
.want = .{ .tag = .table },
.kind = .table,
.build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const c = try b.addContainer(.{ .cell = .{ .head = false, .alignment = .default } }, &.{try str(b, "a")});
const r = try b.addContainer(.{ .row = .{ .head = false } }, &.{c});
return blockDoc(b, .table, &.{r});
}
}.f,
.intact = struct {
// Built with no header row, so a header row in the reparse is one the
// conversion invented.
fn f(back: *const AST) bool {
for (back.nodes) |n| switch (n.kind) {
.row => |r| if (r.head) return false,
else => {},
};
return true;
}
}.f,
},
// The same table with a COLUMN AXIS. Probed separately from `table` so the
// two claims stay independent: the table survives every target and the
// column survives none, and a single probe asserting both would pass on the
// table alone.
.{ .label = "column", .want = .{ .tag = .column }, .kind = .column, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const col = try b.addContainer(.column, &.{});
const c = try b.addContainer(.{ .cell = .{ .head = false, .alignment = .default } }, &.{try str(b, "1")});
const r = try b.addContainer(.{ .row = .{ .head = false } }, &.{c});
return blockDoc(b, .table, &.{ col, r });
}
}.f },
.{ .label = "footnote", .want = .{ .tag = .footnote }, .kind = .{ .footnote = .{ .label = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const p = try b.addContainer(.para, &.{try str(b, "note")});
const def = try b.addContainer(.{ .footnote = .{ .label = "1" } }, &.{p});
const ref = try b.addLeaf(.{ .text_leaf = .{ .kind = .footnote_reference, .text = "1" } });
const body = try b.addContainer(.para, &.{ try str(b, "x"), ref });
return b.addContainer(.doc, &.{ body, def });
}
}.f },
.{ .label = "reference", .want = .{ .tag = .reference }, .kind = .{ .reference = .{ .label = "", .destination = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const def = try b.addLeaf(.{ .reference = .{ .label = "r", .destination = "/u" } });
const l = try b.addContainer(.{ .link = .{ .destination = null, .reference = "r" } }, &.{try str(b, "t")});
const p = try b.addContainer(.para, &.{l});
return b.addContainer(.doc, &.{ p, def });
}
}.f },
.{ .label = "citation", .want = .{ .tag = .citation }, .kind = .{ .citation = .{ .label = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const p = try b.addContainer(.para, &.{try str(b, "Deep Thought.")});
const def = try b.addContainer(.{ .citation = .{ .label = "CIT2002" } }, &.{p});
const ref = try b.addLeaf(.{ .text_leaf = .{ .kind = .citation_reference, .text = "CIT2002" } });
const body = try b.addContainer(.para, &.{ try str(b, "see "), ref });
return b.addContainer(.doc, &.{ body, def });
}
}.f },
// The one kind every target drops. The body is an inline run, not blocks —
// that is the shape twig did not have before it (see `Kind.substitution`).
.{ .label = "substitution", .want = .{ .tag = .substitution }, .kind = .{ .substitution = .{ .label = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const def = try b.addContainer(.{ .substitution = .{ .label = "RST" } }, &.{try str(b, "reStructuredText")});
const ref = try b.addLeaf(.{ .text_leaf = .{ .kind = .substitution_reference, .text = "RST" } });
const body = try b.addContainer(.para, &.{ ref, try str(b, " rules") });
return b.addContainer(.doc, &.{ body, def });
}
}.f },
.{ .label = "container(block)", .want = .{ .container_named = "note" }, .kind = .{ .container = .{ .name = "note", .form = .block_fenced } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const p = try b.addContainer(.para, &.{try str(b, "x")});
return blockDoc(b, .{ .container = .{ .name = "note", .form = .block_fenced } }, &.{p});
}
}.f },
.{ .label = "container(inline)", .want = .{ .container_named = "span" }, .kind = .{ .container = .{ .name = "span", .form = .inline_text } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return inlineDoc(b, .{ .container = .{ .name = "span", .form = .inline_text } }, &.{try str(b, "x")});
}
}.f },
// The ANONYMOUS container, probed apart from the two named ones because
// the answer differs by instance rather than by kind: djot's own divs
// carry no name, so there is nothing for djot to lose and `:::` returns as
// itself — while the named probes above measure a name djot can only hold
// as a class. One `.container` row could not have said both.
.{ .label = "container(anonymous)", .want = .{ .container_named = "" }, .kind = .{ .container = .{ .name = "", .form = .block_fenced } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const p = try b.addContainer(.para, &.{try str(b, "x")});
return blockDoc(b, .{ .container = .{ .name = "", .form = .block_fenced } }, &.{p});
}
}.f },
.{ .label = "comment", .want = .{ .markup_leaf = .comment }, .kind = .{ .markup_leaf = .{ .kind = .comment, .text = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return blockDoc(b, .{ .markup_leaf = .{ .kind = .comment, .text = " hi " } }, &.{});
}
}.f },
.{ .label = "doctype", .want = .{ .markup_leaf = .doctype }, .kind = .{ .markup_leaf = .{ .kind = .doctype, .text = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return blockDoc(b, .{ .markup_leaf = .{ .kind = .doctype, .text = "html" } }, &.{});
}
}.f },
.{ .label = "cdata", .want = .{ .markup_leaf = .cdata }, .kind = .{ .markup_leaf = .{ .kind = .cdata, .text = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return blockDoc(b, .{ .markup_leaf = .{ .kind = .cdata, .text = "raw" } }, &.{});
}
}.f },
.{ .label = "processing_instruction", .want = .{ .tag = .processing_instruction }, .kind = .{ .processing_instruction = .{ .target = "", .data = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return blockDoc(b, .{ .processing_instruction = .{ .target = "php", .data = "echo" } }, &.{});
}
}.f },
.{ .label = "link", .want = .{ .tag = .link }, .kind = .{ .link = .{ .destination = null, .reference = null } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return inlineDoc(b, .{ .link = .{ .destination = "/u", .reference = null } }, &.{try str(b, "t")});
}
}.f },
.{ .label = "image", .want = .{ .tag = .image }, .kind = .{ .image = .{ .destination = null, .reference = null } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
return inlineDoc(b, .{ .image = .{ .destination = "/u.png", .reference = null } }, &.{try str(b, "alt")});
}
}.f },
.{ .label = "raw_inline", .want = .{ .tag = .raw_inline }, .kind = .{ .raw_inline = .{ .format = "", .text = "" } }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const n = try b.addLeaf(.{ .raw_inline = .{ .format = "html", .text = "<b>" } });
return b.addContainer(.doc, &.{try b.addContainer(.para, &.{n})});
}
}.f },
.{ .label = "hard_break", .want = .{ .tag = .hard_break }, .kind = .hard_break, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const n = try b.addLeaf(.hard_break);
const p = try b.addContainer(.para, &.{ try str(b, "a"), n, try str(b, "b") });
return b.addContainer(.doc, &.{p});
}
}.f },
.{ .label = "non_breaking_space", .want = .{ .tag = .non_breaking_space }, .kind = .non_breaking_space, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const n = try b.addLeaf(.non_breaking_space);
const p = try b.addContainer(.para, &.{ try str(b, "a"), n, try str(b, "b") });
return b.addContainer(.doc, &.{p});
}
}.f },
.{ .label = "smart_punctuation", .want = .{ .tag = .smart_punctuation }, .kind = .{ .smart_punctuation = .ellipses }, .build = struct {
fn f(b: *AST.Builder) anyerror!Node.Id {
const n = try b.addLeaf(.{ .smart_punctuation = .ellipses });
return b.addContainer(.doc, &.{try b.addContainer(.para, &.{n})});
}
}.f },
};
test "every inline mark and text leaf is probed against the table" {
const allocator = testing.allocator;
inline for (comptime std.enums.values(AST.InlineMark)) |m| {
var b = AST.Builder.init(allocator);
defer b.deinit();
const root = try inlineDoc(&b, .{ .inline_mark = m }, &.{try str(&b, "x")});
var ast = try b.finish(root);
defer ast.deinit();
for (round_trippable) |target| {
const claimed = fidelity(target, .{ .inline_mark = m });
const observed = try survives(allocator, &ast, target, .{ .mark = m }, null);
if ((claimed == .faithful) != observed) {
std.debug.print("\nfidelity({s}, mark {s}) claims .{s}, observed survives={}\n", .{ @tagName(target), @tagName(m), @tagName(claimed), observed });
return error.FidelityTableStale;
}
}
}
const leaf_text = std.StaticStringMap([]const u8).initComptime(.{
.{ "symb", "name" },
.{ "verbatim", "c" },
.{ "inline_math", "a+b" },
.{ "display_math", "a+b" },
.{ "url", "https://e.com" },
.{ "email", "a@e.com" },
.{ "footnote_reference", "1" },
.{ "citation_reference", "CIT1" },
.{ "substitution_reference", "RST" },
});
inline for (comptime std.enums.values(AST.TextLeafKind)) |k| {
var b = AST.Builder.init(allocator);
defer b.deinit();
const n = try b.addLeaf(.{ .text_leaf = .{ .kind = k, .text = leaf_text.get(@tagName(k)).? } });
const root = try b.addContainer(.doc, &.{try b.addContainer(.para, &.{n})});
var ast = try b.finish(root);
defer ast.deinit();
for (round_trippable) |target| {
const claimed = fidelity(target, .{ .text_leaf = .{ .kind = k, .text = "" } });
const observed = try survives(allocator, &ast, target, .{ .text_leaf = k }, null);
if ((claimed == .faithful) != observed) {
std.debug.print("\nfidelity({s}, leaf {s}) claims .{s}, observed survives={}\n", .{ @tagName(target), @tagName(k), @tagName(claimed), observed });
return error.FidelityTableStale;
}
}
}
}
// ── the attribute probe ────────────────────────────────────────────────────
/// The four keys every probed node carries: one per class the table
/// distinguishes, and a second "other" key to catch a format that treats
/// `title` specially where the table says it does not.
const attr_probe_keys = [_]AST.KeyVal{
.{ .key = "id", .value = "idv" },
.{ .key = "class", .value = "clsv" },
.{ .key = "data-k", .value = "datav" },
.{ .key = "title", .value = "titlev" },
};
/// The first node in a builder whose kind tag matches `kind`'s — `probedNode`
/// before `finish`, so the probe can set attributes on it.
fn probedBuilderNode(b: *const AST.Builder, kind: Node.Kind) ?Node.Id {
for (b.nodes.items, 0..) |n, i| {
if (std.meta.activeTag(n.kind) == std.meta.activeTag(kind)) return @intCast(i);
}
return null;
}
/// What one key of the probe did on a round-trip: on the node it started on,
/// somewhere in the output the reparse did not read as that, or nowhere.
fn observedAttrFidelity(back: *const AST, node: ?Node.Id, src: []const u8, kv: AST.KeyVal) Fidelity {
const value = kv.value.?;
if (node) |id| {
if (back.attrsOf(id).get(kv.key)) |v| {
// A class can be merged with one the target adds (a name carried
// as a class), so it is looked for rather than compared.
const kept = if (std.mem.eql(u8, kv.key, "class")) std.mem.indexOf(u8, v, value) != null else std.mem.eql(u8, v, value);
if (kept) return .faithful;
}
}
return if (std.mem.indexOf(u8, src, value) != null) .degraded else .dropped;
}
/// The attribute half of the probe over one built document: for every target
/// that keeps the node, each of the four keys must do what `attrsFidelity`
/// says.
fn expectAttrsMeasured(allocator: Allocator, label: []const u8, ast: *const AST, kind: Node.Kind, want: AST.KindRef, stale: *bool) !void {
for (round_trippable) |target| {
if (nodeFidelity(target, ast, probedNode(ast, kind).?) != .faithful) continue;
const src = try format.targetEntryFor(target).serializeFromAst.?(allocator, ast);
defer allocator.free(src);
const cfg = format.ParseConfig{};
var back = try format.entryFor(target.asFormat().?).parseToAst(&cfg, allocator, src);
defer back.deinit();
var found: ?Node.Id = null;
for (back.ast.nodes, 0..) |n, i| {
if (want.matches(n.kind)) {
found = @intCast(i);
break;
}
}
const table = attrsFidelity(target, kind);
for (attr_probe_keys) |kv| {
const claimed = table.forKey(kv.key);
const observed = observedAttrFidelity(&back.ast, found, src, kv);
if (claimed != observed) {
std.debug.print(
"\nattrsFidelity({s}, {s}) claims .{s} for `{s}`, but a round-trip observes .{s}\n--- output ---\n{s}\n",
.{ @tagName(target), label, @tagName(claimed), kv.key, @tagName(observed), src },
);
stale.* = true;
}
}
}
}
test "the attribute table matches what the serializers actually do" {
const allocator = testing.allocator;
const attrs: AST.Attrs = .{ .entries = &attr_probe_keys };
// Every mismatch is printed before the test fails, so one run reads the
// whole difference between the table and the serializers.
var stale = false;
// A paragraph is not among the kind probes — every inline probe already
// sits in one — but it is the block the attribute question is asked of
// first.
{
var b = AST.Builder.init(allocator);
defer b.deinit();
const root = try blockDoc(&b, .para, &.{try str(&b, "x")});
try b.setAttrs(probedBuilderNode(&b, .para).?, attrs);
var ast = try b.finish(root);
defer ast.deinit();
try expectAttrsMeasured(allocator, "para", &ast, .para, .{ .tag = .para }, &stale);
}
for (probes) |p| {
var b = AST.Builder.init(allocator);
defer b.deinit();
const root = try p.build(&b);
try b.setAttrs(probedBuilderNode(&b, p.kind).?, attrs);
var ast = try b.finish(root);
defer ast.deinit();
try expectAttrsMeasured(allocator, p.label, &ast, p.kind, p.want, &stale);
}
inline for (comptime std.enums.values(AST.InlineMark)) |m| {
var b = AST.Builder.init(allocator);
defer b.deinit();
const root = try inlineDoc(&b, .{ .inline_mark = m }, &.{try str(&b, "x")});
try b.setAttrs(probedBuilderNode(&b, .{ .inline_mark = m }).?, attrs);
var ast = try b.finish(root);
defer ast.deinit();
try expectAttrsMeasured(allocator, "mark " ++ @tagName(m), &ast, .{ .inline_mark = m }, .{ .mark = m }, &stale);
}
const leaf_text = std.StaticStringMap([]const u8).initComptime(.{
.{ "symb", "name" },
.{ "verbatim", "c" },
.{ "inline_math", "a+b" },
.{ "display_math", "a+b" },
.{ "url", "https://e.com" },
.{ "email", "a@e.com" },
.{ "footnote_reference", "1" },
.{ "citation_reference", "CIT1" },
.{ "substitution_reference", "RST" },
});
inline for (comptime std.enums.values(AST.TextLeafKind)) |k| {
var b = AST.Builder.init(allocator);
defer b.deinit();
const n = try b.addLeaf(.{ .text_leaf = .{ .kind = k, .text = leaf_text.get(@tagName(k)).? } });
const root = try b.addContainer(.doc, &.{try b.addContainer(.para, &.{n})});
try b.setAttrs(n, attrs);
var ast = try b.finish(root);
defer ast.deinit();
const kind: Node.Kind = .{ .text_leaf = .{ .kind = k, .text = "" } };
try expectAttrsMeasured(allocator, "leaf " ++ @tagName(k), &ast, kind, .{ .text_leaf = k }, &stale);
}
if (stale) return error.AttributeTableStale;
}
test "a djot paragraph's class converted to Markdown is reported, naming the key" {
// The class rides on a `<div>` around the paragraph, which Markdown's
// default parser reads as raw HTML beside it: written, not read back.
const Djot = @import("languages/djot/djot.zig");
var doc = try Djot.parse(testing.allocator, "{.center data-size=\"large\"}\nhello\n");
defer doc.deinit();
var arena = std.heap.ArenaAllocator.init(testing.allocator);
defer arena.deinit();
const md = try analyze(arena.allocator(), &doc.ast, doc.ast.root, .markdown);
try testing.expectEqual(@as(usize, 1), md.len);
try testing.expectEqual(Warning.Subject.attrs, md[0].subject);
try testing.expectEqual(Fidelity.degraded, md[0].fidelity);
try testing.expectEqualStrings("para", md[0].kind);
try testing.expectEqual(@as(usize, 2), md[0].attrs.len);
try testing.expectEqualStrings("class", md[0].attrs[0]);
try testing.expectEqualStrings("data-size", md[0].attrs[1]);
var out: Writer.Allocating = .init(testing.allocator);
defer out.deinit();
try md[0].render(&out.writer, .markdown);
try testing.expectEqualStrings(
"`para` at `0` carries attributes (class, data-size) that markdown writes where its parser does not read them back",
out.written(),
);
// An inline mark has nowhere for one at all.
var em = try Djot.parse(testing.allocator, "_x_{.big}\n");
defer em.deinit();
const ew = try analyze(arena.allocator(), &em.ast, em.ast.root, .markdown);
try testing.expectEqual(@as(usize, 1), ew.len);
try testing.expectEqual(Fidelity.dropped, ew[0].fidelity);
try testing.expectEqualStrings("emph", ew[0].kind);
out.clearRetainingCapacity();
try ew[0].render(&out.writer, .markdown);
try testing.expectEqualStrings(
"`emph` at `0/0` carries attributes (class) that markdown cannot write; they are dropped",
out.written(),
);
// The same paragraph to HTML and AsciiDoc keeps both keys: nothing to say.
for ([_]Target{ .html, .asciidoc, .djot }) |t| {
const w = try analyze(arena.allocator(), &doc.ast, doc.ast.root, t);
try testing.expectEqual(@as(usize, 0), w.len);
}
}
test "a key the target keeps is not named beside one it loses" {
// AsciiDoc's inline `[#id.role]` keeps an id and a role on emphasis and
// has no slot for anything else — so only the third key is reported.
const Html = @import("languages/html/html.zig");
var doc = try Html.parse(testing.allocator, "<p><em id=\"a\" class=\"b\" data-k=\"v\">x</em></p>");
defer doc.deinit();
var arena = std.heap.ArenaAllocator.init(testing.allocator);
defer arena.deinit();
const w = try analyze(arena.allocator(), &doc.ast, doc.ast.root, .asciidoc);
try testing.expectEqual(@as(usize, 1), w.len);
try testing.expectEqual(Warning.Subject.attrs, w[0].subject);
try testing.expectEqualStrings("emph", w[0].kind);
try testing.expectEqual(@as(usize, 1), w[0].attrs.len);
try testing.expectEqualStrings("data-k", w[0].attrs[0]);
}
test "a lossy node's attributes are reported once, with the node" {
// A named container is degraded in djot (its name becomes a class); its
// attributes go with it and are not reported a second time.
const Html = @import("languages/html/html.zig");
var doc = try Html.parse(testing.allocator, "<video controls src=\"a.mp4\"></video>");
defer doc.deinit();
var arena = std.heap.ArenaAllocator.init(testing.allocator);
defer arena.deinit();
const w = try analyze(arena.allocator(), &doc.ast, doc.ast.root, .djot);
try testing.expect(w.len >= 1);
for (w) |warning| try testing.expectEqual(Warning.Subject.node, warning.subject);
}
test "a degraded attribute renders as written-but-unread" {
// djot writes a heading's attributes on the line before it, where its
// own parser reads a block naming an id as the section's. The table
// answers per kind, so a class-only heading is reported the same way.
const Html = @import("languages/html/html.zig");
var doc = try Html.parse(testing.allocator, "<h2 class=\"x\">t</h2>");
defer doc.deinit();
var arena = std.heap.ArenaAllocator.init(testing.allocator);
defer arena.deinit();
const w = try analyze(arena.allocator(), &doc.ast, doc.ast.root, .djot);
try testing.expectEqual(@as(usize, 1), w.len);
try testing.expectEqual(Fidelity.degraded, w[0].fidelity);
var out: Writer.Allocating = .init(testing.allocator);
defer out.deinit();
try w[0].render(&out.writer, .djot);
try testing.expectEqualStrings(
"`heading` at `0` carries attributes (class) that djot writes where its parser does not read them back",
out.written(),
);
}