Skip to main content

Crate scrapeunblocker

Crate scrapeunblocker 

Source
Expand description

Official Rust client for the ScrapeUnblocker API.

Scrape any page past Cloudflare, DataDome, PerimeterX, Akamai and other anti-bot protection, as rendered HTML, clean Markdown or parsed JSON; run Google searches; fetch images; and call ready-made plugins for sites such as Amazon, Walmart, Google Maps, Zillow and Booking.com.

use scrapeunblocker::{Client, Step};

let su = Client::from_env()?; // reads SCRAPEUNBLOCKER_KEY

// Rendered HTML
let html = su.get_page_source("https://example.com").await?;

// Clean Markdown for an LLM
let md = su.get_markdown("https://en.wikipedia.org/wiki/Web_scraping").await?;

// Structured JSON, no parser to write
let page = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW").proxy_country("us").await?;
println!("{:?} {}", page.page_type, page.data["title"]);

// Google results
let serp = su.serp("best running shoes").pages_to_check(2).await?;

// A plugin: every option is a setter on the request
let results = su
    .amazon_search("wireless headphones")
    .marketplace("amazon.de")
    .sort("price_asc")
    .await?;

// Drive the browser before the HTML is captured
let html = su
    .get_page_source("https://example.com")
    .steps([Step::click("#more"), Step::wait_for(".results")])
    .await?;

Every endpoint method returns a request builder: set options on it, then .await it (or call .send()). Builders own a cheap clone of the client, so a request can be handed to tokio::spawn. Plugin calls return the API’s JSON as a serde_json::Value.

Errors are Error; an error response from the API is Error::Api with an ApiErrorKind. HTTP 429/502/503/504 and network errors are retried twice with backoff (see ClientBuilder::max_retries).

The blocking client (blocking::Client, feature blocking) has the same methods; finish a request with .send().

Re-exports§

pub use serde_json;

Modules§

blockingblocking
A blocking client, for code without an async runtime (feature blocking).
plugins
Generated by headquarters/scripts/sdk-plugins/generate.py from the public OpenAPI spec - do not edit by hand.

Structs§

ApiError
An error response from the API.
CapturedPage
A page plus the XHR/fetch responses it received, from Client::get_page_with_xhr.
Client
Client for the ScrapeUnblocker API.
ClientBuilder
Builds a Client with non-default settings.
GetImage
Request builder for Client::get_image. Await it for the image bytes.
GetMarkdown
Request builder for Client::get_markdown. Await it for the Markdown.
GetPageSource
Request builder for Client::get_page_source. Await it for the HTML.
GetPageWithCookies
Request builder for Client::get_page_with_cookies. Await it for a PageResult.
GetPageWithXhr
Request builder for Client::get_page_with_xhr. Await it for a CapturedPage.
GetParsed
Request builder for Client::get_parsed. Await it for a ParsedPage.
ListElements
Request builder for Client::list_elements. Await it for the JSON.
PageResult
A page plus the cookies and proxy that served it, from Client::get_page_with_cookies.
ParsedPage
Structured data extracted from a page, from Client::get_parsed.
Plugin
Request builder for Client::plugin. Await it for the JSON.
Serp
Request builder for Client::serp. Await it for the SERP JSON.
Step
One browser action for the steps option of Client::get_page_source and Client::get_markdown.
StepFailure
Details of a browser step that could not run.
TargetNotFound
Details of a target page that answered 404 or 410 on its own.
XhrCall
One XHR/fetch response the page received, from Client::get_page_with_xhr.

Enums§

ApiErrorKind
What kind of error response the API sent.
Error
Everything that can go wrong in a call.

Constants§

API_KEY_ENV
The environment variable the API key is read from when none is given.
DEFAULT_BASE_URL
The production API host.
DEFAULT_MAX_RETRIES
Retries for transient failures (HTTP 429/502/503/504 and network errors).
DEFAULT_TIMEOUT
Per-request timeout. Rendering heavy, protected pages can take a while.
VERSION
This crate’s version, sent in the User-Agent header.

Type Aliases§

ResponseFuture
The future a request builder turns into when awaited.
Result
A Result whose error is this crate’s Error.