Skip to main content

Crate domtree

Crate domtree 

Source
Expand description

§dom-tree-rs

A tiny, zero-dependency, forgiving HTML parser. It turns messy real-world HTML into a clean DOM tree (and JSON) and never panics on bad input — ideal for the browser, the edge, or anywhere you don’t want to pull in a 100 KB+ parser.

It is not a spec-compliant WHATWG parser (see html5ever for that) and not a speed record (see tl). It is the smallest way to get a usable tree out of arbitrary HTML.

§Quick start

let dom = domtree::parse(r#"<ul id="list"><li class="item">Item 1</li></ul>"#);

let li = dom.find_by_tag("li").next().unwrap();
assert_eq!(li.attr("class"), Some("item"));
assert_eq!(li.text_content(), "Item 1");

// The first list element:
let ul = dom.root().unwrap();
assert_eq!(ul.tag_name(), Some("ul"));
assert_eq!(ul.children().count(), 1);

§What it handles

Void elements (<br>, <img>, …), self-closing tags, comments, doctypes, CDATA, raw text (<script>/<style>), boolean and unquoted attributes, hyphenated/custom tags, common HTML entities, and the everyday “forgot a closing tag” cases (<li>, <p>, table cells, …) via implicit closing.

§What it deliberately does not do

Full WHATWG tree construction: no adoption-agency algorithm, no automatic <html>/<head>/<body> insertion, no foster parenting. Anything it has to recover from is recorded in Dom::errors.

No unbounded search for the element an end tag closes: it must be among the 256 innermost open elements or the 8 outermost (where </body> and </html> find theirs). An end tag for an element open only in between is ignored as stray (ErrorKind::StrayEndTag) and the element stays open. Real documents almost never nest that deep, and the bound keeps every end tag O(1) on hostile input.

§Getting content out

let dom = domtree::parse(
    "<article><h2>News</h2><p>It <em>works</em>!</p><script>track()</script></article>",
);
assert_eq!(dom.to_text(), "News\n\nIt works!");
assert_eq!(dom.to_markdown(), "## News\n\nIt *works*!");

§Editing and source positions

A Dom doubles as a small HTML rewriter: Dom::remove, Dom::set_text, Dom::set_attr, Dom::rename, Dom::wrap, Dom::unwrap, parsed-fragment insertion (Dom::append_html and friends) and node moves (Dom::append_child, Dom::insert_before). Parsed nodes also remember where they came from: NodeRef::source_range plus line_col.

let src = "<p>Hello <b>world</b></p>";
let mut dom = domtree::parse(src);
let b = dom.find_by_tag("b").next().unwrap();
assert_eq!(b.source_range(), Some(9..21));
assert_eq!(domtree::line_col(src, 9), (1, 10));

let b = b.id();
dom.rename(b, "strong");
dom.insert_html_after(b, "!");
assert_eq!(dom.to_html(), "<p>Hello <strong>world</strong>!</p>");

§Structure and data

  • Dom::main_content — the article / post / docs body without the navigation, sidebars, comments and footer (“reader mode”).
  • Dom::links / Dom::resources — every link (with its text) and every embedded URL, resolved against the page URL and <base href> (resolve_url, resolve_srcset: RFC 3986, zero-dependency).
  • Dom::metadata — title, description, canonical URL, language, OpenGraph / Twitter-card fields, JSON-LD blocks and feeds.
  • NodeRef::table_rows — a <table> as rows of cell text with colspan/rowspan expanded; to_csv formats them.
let dom = domtree::parse(r#"<html lang="en"><head><title>Release notes</title></head>
  <body><nav><a href="/">Home</a></nav>
  <article><h1>Version 2</h1>
    <p>This release adds link extraction, page metadata, tables as data and a
       reader mode, all without a single new dependency.</p>
    <p>See the <a href="changelog.html">full changelog</a> for details.</p>
  </article></body></html>"#);

assert_eq!(dom.metadata().title.as_deref(), Some("Release notes"));
let article = dom.main_content().unwrap();
assert_eq!(article.tag_name(), Some("article"));
let links = dom.links(Some("https://example.com/news/"));
assert_eq!(links[1].url, "https://example.com/news/changelog.html");
assert_eq!(links[1].text, "full changelog");

§Features

  • json (default) — the zero-dependency Dom::to_json serializer.
  • markdown (default) — the zero-dependency Dom::to_markdown and Dom::to_markdown_chunks.
  • serde — optional serde::Serialize for the tree, Metadata and Chunk (adds the serde dep).
  • query — a zero-dependency CSS selector engine (Dom::select, Selector).
  • entities-full — decode all 2,125 WHATWG named character references instead of the curated ~180 (a sorted static table, about 70 KB).
  • simd — scan text and attribute values with memchr (SIMD, no_std) instead of the built-in word-at-a-time search. Adds that one dependency; tokens and trees are identical either way. Worth it only for very text-heavy input.
  • std — adds std-only conveniences; the crate is no_std + alloc by default.

Structs§

Chunkmarkdown
A piece of a document’s Markdown, small enough for an embedding model or a prompt, with the headings it sits under — so every chunk stands on its own. Built by Dom::to_markdown_chunks.
Descendants
Pre-order descendant iterator (see NodeRef::descendants).
Dom
A parsed HTML document: an arena of nodes, a single string buffer they reference, the top-level (root) nodes, and any recovery diagnostics.
Events
The iterator returned by events. Yields each Event with its byte range in the source.
Link
A hyperlink — an <a href> or <area href> — found by Dom::links.
Metadata
Metadata about a page, from its <title>, <meta> and <link> tags, JSON-LD blocks and a few in-page conventions. Built by Dom::metadata.
NodeId
A handle to a node inside a Dom. Cheap (Copy) and stable for the lifetime of the tree.
NodeRef
A lightweight, Copy read handle to a single node, bound to its Dom.
ParseError
A single recoverable problem encountered while parsing, with the byte offset into the source where it occurred.
Resource
A URL-valued attribute that loads something into the page — an image, script, stylesheet, media file or frame — found by Dom::resources.
Selectorquery
A parsed, reusable CSS selector list.
SelectorErrorquery
Why Selector::parse rejected a selector, and where.
Tokenizer
A streaming HTML tokenizer. Construct with Tokenizer::new (or the top-level crate::tokenize) and iterate to get Tokens.

Enums§

AttrValue
The value side of an attribute.
ErrorKind
The category of a ParseError.
Event
One step of a document as the tree builder sees it, after recovery. See events.
NodeKind
What a node is — a borrowed, Copy view returned by NodeRef::kind.
Token
A single lexical token of HTML.

Traits§

AsSelectorquery
Anything the selector methods accept: a &str or &String (parsed on the spot) or an already-parsed &Selector.

Functions§

events
Stream a document as Events — start and end of elements, text, comments, doctypes — without building a tree.
line_col
The 1-based (line, column) of byte offset byte in src, for error messages, linters and editors — pair it with NodeRef::source_range or ParseError::position.
parse
Parse a full HTML document into a best-effort tree.
parse_fragment
Parse an HTML fragment.
resolve_srcset
Resolve every URL in a srcset attribute value ("a.png 1x, b.png 2x") against base with resolve_url, keeping each candidate’s descriptor.
resolve_url
Resolve reference against base, following RFC 3986 §5.2 (the algorithm browsers use for href and src).
to_csv
Format rows as CSV (RFC 4180): fields separated by commas, a field quoted (with " doubled) when it contains a comma, a quote or a line break, and every record ended by \r\n. Rows may have different lengths.
tokenize
Create a streaming Tokenizer over html without building a tree.