html5-parser 0.1.0

A pure-Rust WHATWG HTML5 tokenizer and tree-construction implementation
Documentation
  • Coverage
  • 55.88%
    19 out of 34 items documented0 out of 1 items with examples
  • Size
  • Source code size: 609.6 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 2.42 MB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 2s Average build duration of successful builds.
  • all releases: 2s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • casoon/html5-parser
    0 0 2
  • crates.io
  • Dependencies
    • —
  • Versions
  • Owners
  • casoon

html5-parser

A pure-Rust WHATWG HTML5 tokenizer and tree-construction implementation.

Scope

Deliberately staged, not built as a fully generic library from day one:

  1. Step 1: only what html-conform (a sibling project) actually needs to replace its current HTML5 parsing dependency — a tokenizer and tree-construction implementation whose output can directly feed html-conform's src/infoset.rs::normalize(), including per-node source positions (its current dependency has none). No generic public API commitment yet.
  2. Step 2: once step 1 is proven against html-conform's real usage, extract the generic, html-conform-agnostic part (a reusable WHATWG HTML5 tokenizer/tree-builder) as this crate's public API.

Architecture (working title)

HTML input (string) → tokenizer (WHATWG tokenizer state machine)
                     → tree_builder (WHATWG tree-construction algorithm,
                       incl. foreign content / SVG / MathML)
                     → document (element/text/comment tree with positions)

Normative basis

Implementation decisions are derived from the WHATWG HTML parsing specification. Other implementations (e.g. html5ever) are explanatory references only, not a source to copy code from.

Status

The tokenizer (§13.2.5) and tree-construction algorithm (§13.2.6 — all insertion modes, adoption agency, foster parenting, foreign content) are implemented and wired end to end.

pub fn parse(input: &str) -> Document and the read-only tree types (Document, NodeId, NodeKind, Node, Attribute, Position, Children) are public — just enough to walk the resulting tree and read each node's kind and source position, matching what html-conform's src/infoset.rs::normalize() needs. Tokenizer/TreeBuilder and everything else stay crate-internal (see Scope above — no commitment to a generic public API yet, that's Step 2); not published to crates.io.

Known limitations

Three gaps were tracked here, evidence-based (per Step 1's scope, above) rather than oversights, closed out one by one ahead of a first crates.io publish (see plan/DECISIONS.md), independently of whether html-conform itself ends up needing each one — all three are now done:

  • Frameset-related insertion modes not implemented — done: "in frameset"/"after frameset"/"after after frameset" (§13.2.6.4.18-21) are implemented, including <frameset> correctly replacing (rather than nesting under) <body> per §13.2.6.4.7/.4.6's real rules. See plan/04-frameset.md.
  • <template> treated as an ordinary element — done (the classic content model): <template> gets a real, separate NodeKind::DocumentFragment "template contents" (§13.2.6.4.4/.16, the stack of template insertion modes, the active-formatting-elements marker). Still not implemented within this: two much newer, still-evolving sub-features layered onto <template> in the current spec, neither exercised by the html5lib-tests corpus: declarative shadow DOM (shadowrootmode and friends) and content patching (the for attribute) — both would require modeling shadow roots/custom element registries this crate has no other use for. See plan/05-template.md.
  • <selectedcontent> treated as an ordinary element — done, with a scope note: this isn't actually a tree-construction (§13.2.6) feature at all — it's the <option> element's own HTML-parser-specific hook (form-elements.html §4.10.10/.17, "maybe clone an option into selectedcontent", run when an option is popped off the stack of open elements), simplified to this crate's parse-time-only, non-scripted needs (no live selectedness mutation, no multiple <select> support, a practically- rather than fully-generally-scoped "list of options"/disabled-flag walk). See plan/06-selectedcontent.md.

Testing

Besides hand-written unit/end-to-end tests (src/tokenizer.rs, src/tree_builder.rs, src/lib.rs, src/document.rs), cargo test also runs tests/html5lib_conformance.rs: every applicable case (full-document, non-fragment, non-scripting — see tests/html5lib-tests/README.md) from the vendored html5lib-tests tree-construction corpus — currently all 1,726 applicable cases pass (100%). tests/html5lib_known_failures.txt is currently empty (kept, rather than deleted, as the harness's regression-tracking mechanism — see that file's header — for whenever a future corpus refresh or code change introduces a real one).

License

MIT — see LICENSE.