domq 0.5.0

Tiny, zero-dependency, forgiving HTML parser: turn messy real-world HTML into a clean DOM tree (and JSON). WASM-first, no_std-friendly.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

# Reader mode, measured

How well `Dom::main_content()` (`domq --main`, the `main` option of the MCP
tools) finds a page's article, and `Dom::metadata()` its metadata, scored
against [Mozilla Readability](https://github.com/mozilla/readability)'s
130 labelled test pages at commit
[`ab4027a`](https://github.com/mozilla/readability/tree/ab4027a8b37669745016869a37a504727992b2ba/test/test-pages).
The reference is what Readability (Firefox's reader view) extracts, so these
numbers measure agreement with it, not ground truth.

<!-- Generated by conformance/ (`cargo run --release --manifest-path
conformance/Cargo.toml --bin reader -- --write docs/reader-mode.md`). Do not
edit: CI regenerates it and fails when it differs. -->

## Main content

Words of domq's pick against the words of Readability's article, both as
`to_text`, compared as bags of lower-cased words. **Precision**: how much of
what domq kept is article; **recall**: how much of the article domq kept.
Means over the pages.

| Extraction | Precision | Recall | F1 |
|---|---:|---:|---:|
| `main_content()` | 93.4% | 98.9% | 95.2% |
| Whole page (no reader mode) | 74.8% | 99.9% | 82.1% |

| Pages where reader mode's F1 is… | Pages |
|---|---:|
| at least 90% | 116 |
| 50–90% | 11 |
| below 50% | 3 |

On 3 pages `main_content()` found no container and the whole page
was used.

## Metadata

`Dom::metadata()` against Readability's metadata, on the pages where
Readability found a value; equal after collapsing whitespace.

| Field | Pages | Same |
|---|---:|---:|
| title | 127 | 48.8% |
| author (byline) | 76 | 36.8% |
| site name | 62 | 85.5% |
| language | 73 | 100.0% |
| published time | 33 | 48.5% |

Readability's byline also comes from the page text ("By Jane Doe"), and its
title drops a trailing site name ("Article | Site"); domq reads both from
markup only, so those two rows count differences of convention too.

## Hardest pages

The 15 pages with the lowest reader-mode F1, for tuning the heuristic:

| Page | Precision | Recall | F1 |
|---|---:|---:|---:|
| `bug-1255978` | 17.4% | 100.0% | 29.7% |
| `yahoo-3` | 21.5% | 100.0% | 35.4% |
| `nytimes-5` | 23.3% | 100.0% | 37.8% |
| `hukumusume` | 100.0% | 50.5% | 67.1% |
| `bbc-1` | 53.1% | 100.0% | 69.3% |
| `tmz-1` | 65.4% | 100.0% | 79.1% |
| `wordpress` | 67.5% | 100.0% | 80.6% |
| `yahoo-4` | 70.1% | 100.0% | 82.4% |
| `herald-sun-1` | 100.0% | 73.1% | 84.5% |
| `heise` | 75.7% | 100.0% | 86.2% |
| `theverge` | 77.8% | 100.0% | 87.5% |
| `nytimes-4` | 80.9% | 95.3% | 87.5% |
| `qq` | 91.1% | 85.0% | 87.9% |
| `mozilla-1` | 79.8% | 100.0% | 88.8% |
| `gitlab-blog` | 82.7% | 100.0% | 90.5% |

The test pages are © their publishers, collected by the Readability project
(Apache-2.0); they are downloaded by `conformance/fetch.sh`, not stored in
this repository.