Expand description
Official Rust client for the ScrapeUnblocker API.
Scrape any page past Cloudflare, DataDome, PerimeterX, Akamai and other anti-bot protection, as rendered HTML, clean Markdown or parsed JSON; run Google searches; fetch images; and call ready-made plugins for sites such as Amazon, Walmart, Google Maps, Zillow and Booking.com.
use scrapeunblocker::{Client, Step};
let su = Client::from_env()?; // reads SCRAPEUNBLOCKER_KEY
// Rendered HTML
let html = su.get_page_source("https://example.com").await?;
// Clean Markdown for an LLM
let md = su.get_markdown("https://en.wikipedia.org/wiki/Web_scraping").await?;
// Structured JSON, no parser to write
let page = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW").proxy_country("us").await?;
println!("{:?} {}", page.page_type, page.data["title"]);
// Google results
let serp = su.serp("best running shoes").pages_to_check(2).await?;
// A plugin: every option is a setter on the request
let results = su
.amazon_search("wireless headphones")
.marketplace("amazon.de")
.sort("price_asc")
.await?;
// Drive the browser before the HTML is captured
let html = su
.get_page_source("https://example.com")
.steps([Step::click("#more"), Step::wait_for(".results")])
.await?;Every endpoint method returns a request builder: set options on it, then
.await it (or call .send()). Builders own a cheap clone of the client,
so a request can be handed to tokio::spawn. Plugin calls return the
API’s JSON as a serde_json::Value.
Errors are Error; an error response from the API is Error::Api with
an ApiErrorKind. HTTP 429/502/503/504 and network errors are retried
twice with backoff (see ClientBuilder::max_retries).
The blocking client (blocking::Client, feature blocking) has the same
methods; finish a request with .send().
Re-exports§
pub use serde_json;
Modules§
- blocking
blocking - A blocking client, for code without an async runtime (feature
blocking). - plugins
- Generated by headquarters/scripts/sdk-plugins/generate.py from the public OpenAPI spec - do not edit by hand.
Structs§
- ApiError
- An error response from the API.
- Captured
Page - A page plus the XHR/fetch responses it received, from
Client::get_page_with_xhr. - Client
- Client for the ScrapeUnblocker API.
- Client
Builder - Builds a
Clientwith non-default settings. - GetImage
- Request builder for
Client::get_image. Await it for the image bytes. - GetMarkdown
- Request builder for
Client::get_markdown. Await it for the Markdown. - GetPage
Source - Request builder for
Client::get_page_source. Await it for the HTML. - GetPage
With Cookies - Request builder for
Client::get_page_with_cookies. Await it for aPageResult. - GetPage
With Xhr - Request builder for
Client::get_page_with_xhr. Await it for aCapturedPage. - GetParsed
- Request builder for
Client::get_parsed. Await it for aParsedPage. - List
Elements - Request builder for
Client::list_elements. Await it for the JSON. - Page
Result - A page plus the cookies and proxy that served it, from
Client::get_page_with_cookies. - Parsed
Page - Structured data extracted from a page, from
Client::get_parsed. - Plugin
- Request builder for
Client::plugin. Await it for the JSON. - Serp
- Request builder for
Client::serp. Await it for the SERP JSON. - Step
- One browser action for the
stepsoption ofClient::get_page_sourceandClient::get_markdown. - Step
Failure - Details of a browser step that could not run.
- Target
NotFound - Details of a target page that answered 404 or 410 on its own.
- XhrCall
- One XHR/fetch response the page received, from
Client::get_page_with_xhr.
Enums§
- ApiError
Kind - What kind of error response the API sent.
- Error
- Everything that can go wrong in a call.
Constants§
- API_
KEY_ ENV - The environment variable the API key is read from when none is given.
- DEFAULT_
BASE_ URL - The production API host.
- DEFAULT_
MAX_ RETRIES - Retries for transient failures (HTTP 429/502/503/504 and network errors).
- DEFAULT_
TIMEOUT - Per-request timeout. Rendering heavy, protected pages can take a while.
- VERSION
- This crate’s version, sent in the
User-Agentheader.
Type Aliases§
- Response
Future - The future a request builder turns into when awaited.
- Result
- A
Resultwhose error is this crate’sError.