readabilityrs/
lib.rs

1//! # readabilityrs
2//!
3//! Pulls the article out of a web page. This is a Rust port of
4//! [Mozilla's Readability.js](https://github.com/mozilla/readability), the algorithm
5//! behind Firefox Reader View: give it a page of HTML and it returns the title,
6//! byline, body, excerpt, site name, language, and publication time, leaving
7//! navigation, ads, and related-article rails behind.
8//!
9//! It passes 119 of the 130 cases in Mozilla's test suite. The 11 differences are
10//! editorial rather than failures, and each is named in `tests/mozilla_test_suite.rs`.
11//!
12//! Output is cleaned HTML by default, in [`Article::content`]. Enabling
13//! [`ReadabilityOptions::output_markdown`] additionally produces Markdown, after a
14//! standardization pass that rewrites vendor-specific markup (highlighted code,
15//! lazy-loaded images, footnotes, MathJax and KaTeX output) into canonical form.
16//!
17//! ## Basic Usage
18//!
19//! ```rust,no_run
20//! use readabilityrs::{Readability, ReadabilityOptions};
21//!
22//! let html = r#"<html><body><article><h1>Title</h1><p>Content...</p></article></body></html>"#;
23//! let url = "https://example.com/article";
24//!
25//! let options = ReadabilityOptions::default();
26//! let readability = Readability::new(html, Some(url), Some(options)).unwrap();
27//!
28//! if let Some(article) = readability.parse() {
29//!     println!("Title: {:?}", article.title);
30//!     println!("Content: {:?}", article.content);
31//!     println!("Author: {:?}", article.byline);
32//! }
33//! ```
34//!
35//! ## Advanced Usage
36//!
37//! ### Custom Options
38//!
39//! ```rust,no_run
40//! use readabilityrs::{Readability, ReadabilityOptions};
41//!
42//! let html = "<html>...</html>";
43//!
44//! let options = ReadabilityOptions::builder()
45//!     .char_threshold(300)
46//!     .nb_top_candidates(10)
47//!     .build();
48//!
49//! let readability = Readability::new(html, None, Some(options)).unwrap();
50//! let article = readability.parse();
51//! ```
52//!
53//! ### Pre-flight Check
54//!
55//! Use [`is_probably_readerable`] to quickly check if a document is likely to be parseable
56//! before doing the full parse:
57//!
58//! ```rust,no_run
59//! use readabilityrs::is_probably_readerable;
60//!
61//! let html = "<html>...</html>";
62//!
63//! if is_probably_readerable(html, None) {
64//!     // Proceed with full parsing
65//! } else {
66//!     // Skip parsing or use alternative strategy
67//! }
68//! ```
69//!
70//! ## Error Handling
71//!
72//! ```rust,no_run
73//! use readabilityrs::{Readability, ReadabilityError};
74//!
75//! let html = "<html>...</html>";
76//! let url = "not a valid url";
77//!
78//! match Readability::new(html, Some(url), None) {
79//!     Ok(readability) => {
80//!         if let Some(article) = readability.parse() {
81//!             println!("Success!");
82//!         }
83//!     }
84//!     Err(ReadabilityError::InvalidUrl(url)) => {
85//!         eprintln!("Invalid URL: {}", url);
86//!     }
87//!     Err(e) => {
88//!         eprintln!("Error: {}", e);
89//!     }
90//! }
91//! ```
92//!
93//! ## Security
94//!
95//! [`Article::content`] comes from untrusted input and is **not sanitized by
96//! default**. This matches the Readability.js contract: every attribute of every
97//! element that survives extraction is written back out, including event handlers
98//! such as `onerror` and `onclick`, and URL schemes such as `javascript:` and
99//! `data:text/html`. Anything that renders the output in a webview or browser DOM
100//! has to sanitize it first, for example with
101//! [`ammonia`](https://crates.io/crates/ammonia).
102//!
103//! Setting [`ReadabilityOptions::sanitize_content`] drops script-bearing and
104//! content-loading elements whole, along with event-handler attributes, the
105//! highest-risk URL schemes, and comments. It reduces harm and is not a substitute
106//! for a real sanitizer: the allowed elements keep every other attribute they
107//! carry, and none of it applies to [`Article::markdown_content`].
108//!
109//! ## Algorithm
110//!
111//! Extraction runs in phases. The document is preprocessed first: scripts and styles
112//! are stripped, `<noscript>` wrappers around lazy-loaded images are unwrapped, and
113//! deprecated elements are normalized. Candidate containers are then scored by tag
114//! type, text density, link density, and class and id patterns. The highest-scoring
115//! subtree becomes the article body, and sibling elements that look like part of the
116//! same article are pulled in with it. Post-processing cleans the result.
117//!
118//! When a pass produces too little text, it is retried with looser flags: first
119//! without the unlikely-candidate filter, then without class weighting, then without
120//! conditional cleaning. If every attempt stays under the character threshold, the
121//! longest one is returned.
122
123/// Compiles every `rust` fence in `README.md` as a doctest so the front page of
124/// crates.io cannot drift from the API. Exists only under `cargo test`; the README
125/// is deliberately not pulled into the crate docs, since it carries badges, install
126/// instructions and repository-relative links that do not belong on docs.rs.
127#[cfg(doctest)]
128#[doc = include_str!("../README.md")]
129struct ReadmeDoctests;
130
131mod article;
132mod cleaner;
133mod constants;
134mod content_extractor;
135mod dom_utils;
136pub mod elements;
137mod error;
138pub mod markdown;
139mod metadata;
140mod options;
141mod post_processor;
142mod preformatted;
143mod readability;
144mod readerable;
145mod scoring;
146mod utils;
147
148// Public exports
149pub use article::Article;
150pub use error::{ReadabilityError, Result};
151pub use markdown::MarkdownOptions;
152pub use options::ReadabilityOptions;
153pub use readability::Readability;
154pub use readerable::{is_probably_readerable, ReaderableOptions};
readabilityrs/lib.rs

readabilityrs/
lib.rs
