Crate scraper 

scraper

HTML parsing and querying with CSS selectors.

scraper is on crates.io and GitHub.

Scraper provides an interface to Servo’s html5ever and selectors crates, for browser-grade parsing and querying.

Examples

Parsing a document

use scraper::Html;

let html = r#"
    <!DOCTYPE html>
    <meta charset="utf-8">
    <title>Hello, world!</title>
    <h1 class="foo">Hello, <i>world!</i></h1>
"#;

let document = Html::parse_document(html);

Parsing a fragment

use scraper::Html;
let fragment = Html::parse_fragment("<h1>Hello, <i>world!</i></h1>");

Parsing a selector

use scraper::Selector;
let selector = Selector::parse("h1.foo").unwrap();

Selecting elements

use scraper::{Html, Selector};

let html = r#"
    <ul>
        <li>Foo</li>
        <li>Bar</li>
        <li>Baz</li>
    </ul>
"#;

let fragment = Html::parse_fragment(html);
let selector = Selector::parse("li").unwrap();

for element in fragment.select(&selector) {
    assert_eq!("li", element.value().name());
}

Selecting descendent elements

use scraper::{Html, Selector};

let html = r#"
    <ul>
        <li>Foo</li>
        <li>Bar</li>
        <li>Baz</li>
    </ul>
"#;

let fragment = Html::parse_fragment(html);
let ul_selector = Selector::parse("ul").unwrap();
let li_selector = Selector::parse("li").unwrap();

let ul = fragment.select(&ul_selector).next().unwrap();
for element in ul.select(&li_selector) {
    assert_eq!("li", element.value().name());
}

Accessing element attributes

use scraper::{Html, Selector};

let fragment = Html::parse_fragment(r#"<input name="foo" value="bar">"#);
let selector = Selector::parse(r#"input[name="foo"]"#).unwrap();

let input = fragment.select(&selector).next().unwrap();
assert_eq!(Some("bar"), input.value().attr("value"));

Serializing HTML and inner HTML

use scraper::{Html, Selector};

let fragment = Html::parse_fragment("<h1>Hello, <i>world!</i></h1>");
let selector = Selector::parse("h1").unwrap();

let h1 = fragment.select(&selector).next().unwrap();

assert_eq!("<h1>Hello, <i>world!</i></h1>", h1.html());
assert_eq!("Hello, <i>world!</i>", h1.inner_html());

Accessing descendent text

use scraper::{Html, Selector};

let fragment = Html::parse_fragment("<h1>Hello, <i>world!</i></h1>");
let selector = Selector::parse("h1").unwrap();

let h1 = fragment.select(&selector).next().unwrap();
let text = h1.text().collect::<Vec<_>>();

assert_eq!(vec!["Hello, ", "world!"], text);

Manipulating the DOM

use html5ever::tree_builder::TreeSink;
use scraper::{Html, Selector, HtmlTreeSink};

let html = "<html><body>hello<p class=\"hello\">REMOVE ME</p></body></html>";
let selector = Selector::parse(".hello").unwrap();
let mut document = Html::parse_document(html);
let node_ids: Vec<_> = document.select(&selector).map(|x| x.id()).collect();
let tree = HtmlTreeSink::new(document);
for id in node_ids {
    tree.remove_from_parent(&id);
}
let document = tree.finish();
assert_eq!(document.html(), "<html><head></head><body>hello</body></html>");

Using the library across threads

The html5ever crate uses the Tendril type as their reference counted string. By default, it’s thread-local and thus !Send, enabling the atomic flag will use the atomic counting version of Tendril, implementing Send and allow their use across threads.

[dependencies]
scraper = { version = "0.27.0", features = ["atomic"] }

Contributing

Please feel free to open pull requests. If you’re planning on implementing something big (i.e. not fixing a typo, a small bug fix, minor refactor, etc.) then please open an issue first.

The core types

Parsing produces an Html document that owns a tree of Nodes. Every node is one variant of the Node enum, for example Node::Text for text and Node::Element for an element. The data that belongs to an element node, like its attributes, is held in a node::Element.

Running a Selector over a document does not hand back bare Nodes. CSS selectors only ever match elements, so it yields ElementRefs, each one a handle to an element node that also knows where it sits in the tree. From an ElementRef you can reach the element data with ElementRef::value, read the text under it with ElementRef::text, or select further into its descendants. Because an ElementRef also dereferences to an ego_tree::NodeRef, the tree navigation methods (parent, children, siblings) are available on it too.

The re-exported Element trait is a different thing from node::Element. The trait comes from the selectors crate and is what lets an ElementRef be matched against a CSS selector. Most code never names it directly, though its convenience methods (such as parent_element) are available on an ElementRef through the trait impl, which is why it is re-exported.

use scraper::{Html, Selector};

let document = Html::parse_fragment(r#"<ul><li id="a">one</li><li>two <em>three</em></li></ul>"#);
let selector = Selector::parse("li").unwrap();

for element in document.select(&selector) {
    // The element data: its tag name and attributes.
    let name = element.value().name();
    // `attr` is the short-hand for reading a single attribute by name.
    let id = element.attr("id");
    // All descendant text nodes below this element, concatenated.
    let text = element.text().collect::<String>();
    println!("{name} id={id:?} text={text:?}");
}

Re-exports

pub use crate::element_ref::ElementRef;

pub use crate::html::Html;

pub use crate::html::HtmlTreeSink;

pub use crate::node::Node;

pub use crate::selector::Selector;

Modules

element_ref

Element references.

error

Custom error types for diagnostics Includes re-exported error types from dependencies

html

HTML documents and fragments.

node

HTML nodes.

selectable

Provides the Selectable to abstract over collections of elements

selector

CSS selectors.

Enums

CaseSensitivity

Traits

Element

Type Aliases

StrTendril

Primary string tendril type.