scrapeunblocker 0.20.0

Official Rust client for the ScrapeUnblocker API - scrape any page past Cloudflare, DataDome, PerimeterX and Akamai as HTML, Markdown or parsed JSON.
Documentation

ScrapeUnblocker Rust client

Official Rust client for the ScrapeUnblocker web scraping API.

Scrape any page past Cloudflare, DataDome, PerimeterX, Akamai and other anti-bot protection, as rendered HTML, clean Markdown or parsed JSON; run Google searches; fetch images; and call ready-made plugins for Amazon, Walmart, Google Maps, Zillow, Booking.com and 170+ other sites. Async (Tokio) with an optional blocking client, typed errors and automatic retries.

crates.io docs.rs

Install

cargo add scrapeunblocker
cargo add tokio --features macros,rt-multi-thread

Requires Rust 1.88+.

Quickstart

use scrapeunblocker::Client;

#[tokio::main]
async fn main() -> scrapeunblocker::Result<()> {
    let su = Client::from_env()?; // reads SCRAPEUNBLOCKER_KEY, or Client::new("YOUR_API_KEY")?

    // Rendered HTML for any URL
    let html = su.get_page_source("https://example.com").await?;

    // Structured JSON instead of HTML (products, listings, search results, ...)
    let page = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW").await?;
    println!("{:?} {}", page.page_type, page.data["title"]);
    Ok(())
}

Get your API key at app.scrapeunblocker.com. The free trial does not require a credit card.

Every endpoint method returns a request builder: set options on it, then .await it (or call .send().await). Options you leave out are not sent, so the API's own defaults apply. A builder owns a cheap clone of the client, so a request can be handed straight to tokio::spawn.

Authentication

use std::time::Duration;
use scrapeunblocker::Client;

let su = Client::new("YOUR_API_KEY")?;
let su = Client::from_env()?;            // SCRAPEUNBLOCKER_KEY
let su = Client::builder()
    .api_key("YOUR_API_KEY")
    .timeout(Duration::from_secs(120))   // default 180 s
    .max_retries(3)                      // default 2
    .user_agent("my-crawler/1.4")        // your app's token, sent after the SDK's
    .build()?;

Fetch rendered HTML

let html = su
    .get_page_source("https://example.com")
    .proxy_country("de")      // route through a country
    .time_sleep(2)            // extra seconds after load
    .await?;

Markdown for LLMs

let md = su.get_markdown("https://en.wikipedia.org/wiki/Web_scraping").await?;

// Only the main content by default (no navigation, site header, footer or
// sidebars). Convert the whole page instead:
let full = su.get_markdown(url).main_content_only(false).await?;

More in the Markdown guide.

Browser steps

Run actions in a real browser after the page loads; you get the HTML of the resulting state.

use scrapeunblocker::Step;

let html = su
    .get_page_source("https://example.com")
    .steps([
        Step::type_text("input[name=q]", "wireless headphones"),
        Step::press_key("Enter"),
        Step::wait_for(".search-results").timeout_ms(10_000),
    ])
    .await?;

Step has wait_for, wait_for_text, wait, click, type_text, select, press_key, scroll_to_bottom and scroll_by, plus selector_type, timeout_ms and clear modifiers. Find the selectors first with list_elements:

let page = su.list_elements("https://example.com").await?;
for el in page["elements"].as_array().into_iter().flatten() {
    println!("{} {} {}", el["tag"], el["selector"], el["text"]);
}

Full action reference in the browser steps guide.

Parsed JSON

let page = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW").await?;
println!("{:?} {:?}", page.page_type, page.source);
println!("{}", page.data["price"]);

// If a parse ever comes back wrong, force a fresh set of rules:
let page = su.get_parsed(url).refresh_rules(true).rules_hint("price is missing").await?;

// The page rendered but held no structured data: billed like any page,
// and the HTML is already here
if !page.data_extracted {
    println!("{:?}", page.detail);
    let html = page.html;
}

Google search (SERP)

let serp = su.serp("best running shoes").pages_to_check(2).proxy_country("us").await?;
for result in serp["organic"].as_array().into_iter().flatten() {
    println!("{} {} {}", result["position"], result["title"], result["url"]);
}

.full_urls(true) waits until every result has its real URL (up to about 40 seconds longer, billed the same).

Plugins

Ready-made scrapers for specific sites that return structured JSON (serde_json::Value). Each method takes the plugin's required input positionally; every other parameter is a setter. Every parameter is documented on docs.rs and on the plugin pages.

let products = su.target_search("coffee maker").sort("price_low").max_results(48).await?;
let milk = su.woolworths_product("888137").await?;
let trends = su.google_trends(["coffee", "tea"]).geo("US").await?;      // sent comma-separated
let reviews = su.trustpilot_reviews("www.nike.com").stars([4, 5]).await?;
let flights = su
    .skyscanner_flights()
    .origin("London")
    .dest("New York")
    .depart_date("2026-09-01")
    .currency("USD")
    .await?;
Category Methods
Travel skyscanner_flights, skyscanner_hotels, skyscanner_carhire, booking_hotels, booking_reviews, airbnb_search, airbnb_room, airbnb_reviews, tripadvisor_list, tripadvisor_reviews
Google google_local, google_images, maps_place_details, maps_reviews, google_trends, google_shopping, google_jobs, google_hotels, google_hotels_property, google_news, google_ads_transparency, google_ads_transparency_advertisers, google_ai_mode, google_lens
AI gemini, chatgpt
Search engines bing_search, bing_images, duckduckgo_search, yahoo_search, yahoo_images, brave_search, brave_images
E-commerce amazon_product, amazon_search, amazon_bestsellers, amazon_reviews, amazon_offers, amazon_seller, walmart_search, walmart_product, target_search, target_category, target_product, bestbuy_search, bestbuy_product, homedepot_search, costco_search, costco_product, etsy_search, etsy_product, shein_search, zalando_search, aliexpress_search, aliexpress_product, alibaba_search, alibaba_product, alibaba1688_search, alibaba1688_product, tiktok_shop_search, tiktok_shop_product, tiktok_shop_store, oopbuy_search, smyths_inventory, smyths_details, smyths_search, smyths_category, wish_search, wish_product, shopee_search, shopee_product, taobao_search, taobao_product, flipkart_search, flipkart_product, lazada_search, lazada_product, allegro_search, allegro_product, falabella_search, falabella_product, magazineluiza_search, magazineluiza_product, mediamarkt_search, mediamarkt_product, cdiscount_search, cdiscount_product, idealo_search, idealo_offers, rakuten_search, rakuten_product, indiamart_search, indiamart_product, bedbathandbeyond_search, bedbathandbeyond_product, petco_search, petco_product, neobits_product, lowes_search, lowes_product, menards_search, menards_product, grainger_search, grainger_product
Classifieds & Resale ebay_product, ebay_search, vinted_search, leboncoin_search
Groceries woolworths_search, woolworths_category, woolworths_specials, woolworths_product, coles_search, coles_category, coles_specials, coles_product, aldi_au_search, aldi_au_category, aldi_au_specials, aldi_au_product, kroger_search, kroger_product, instacart_search, instacart_product, publix_search, publix_product, bodegaaurrera_search, bodegaaurrera_product
Real estate zillow_search, zillow_detail, redfin_search, redfin_detail, rightmove_search, idealista_search, idealista_detail, immobilienscout24_search
Vehicles mobile_de, autoscout24_search
Jobs indeed_jobs, glassdoor_jobs, glassdoor_reviews, glassdoor_salaries, naukri_jobs, wellfound_jobs, wellfound_job, wellfound_company, wellfound_company_search, career_site_jobs, upwork_jobs
Reviews & Companies trustpilot_reviews, g2_reviews, g2_search, capterra_reviews, capterra_search, product_hunt, crunchbase_company, crunchbase_search
Local & Maps yelp_search, yelp_business, yelp_reviews, naver_place_search, naver_place_details
Apps google_play_app, google_play_search, app_store_app, app_store_search, app_store_reviews
Video & Music youtube_channel, youtube_search, youtube_transcript, spotify_search, spotify_artist, spotify_album, spotify_track, spotify_playlist
Sports sofascore_events, sofascore_match, sofascore_search, sofascore_team, sofascore_player, sofascore_tournament
SEO & Ads similarweb_overview, ahrefs, semrush_overview, meta_ad_library

Any plugin endpoint can also be called by path, with parameters named as the API names them - handy for an endpoint added after your SDK version:

let data = su
    .plugin("/marketplace/target-search")
    .param("query", "tv")
    .param("max_results", 24)
    .await?;

Cookies and the serving proxy

let page = su.get_page_with_cookies("https://example.com").await?;
println!("{:?} {} {:?}", page.html, page.cookies, page.proxy);

The page's own API calls (XHR/fetch)

Many sites render from a JSON API the page calls itself, and that API often refuses direct requests. get_page_with_xhr renders the page in a real browser and returns every XHR/fetch response whose URL contains the fragment you pass, in the order the page received them. Add steps to make the page fire the call (a search, "load more").

use scrapeunblocker::Step;

let page = su
    .get_page_with_xhr("https://quotes.toscrape.com/scroll", "/api/quotes")
    .steps([Step::scroll_to_bottom(), Step::wait(1500)])
    .await?;
for call in &page.xhr {
    if call.status == 200 {
        println!("{} {}", call.url, call.json()?["quotes"][0]["text"]);
    }
}

Each XhrCall has url, method, status, content_type, body (text) and body_omitted (why there is no body). Up to 20 responses and 10 MB of bodies; page.xhr_truncated is true when more matched. When the site answers every matching call with 403 or 429, the request fails like a block and is not billed.

Images

let bytes = su.get_image("https://example.com/photo.jpg").await?;
std::fs::write("photo.jpg", bytes)?;

The bytes come back exactly as the image host serves them, in the original format (JPEG, PNG, WebP, AVIF, GIF, SVG, ...). Pass the image URL itself, not the page it sits on.

Concurrency

Plan limits are concurrency: how many requests run at the same time on a key. Cap your own in-flight requests to match, e.g. with futures::stream::iter(urls).map(...).buffer_unordered(N) or a tokio::sync::Semaphore; a request over the limit is answered 429 and retried with backoff.

Blocking client

Enable the blocking feature for code without an async runtime. Same methods, finish a request with .send():

scrapeunblocker = { version = "0.20", features = ["blocking"] }
use scrapeunblocker::blocking::Client;

let su = Client::from_env()?;
let html = su.get_page_source("https://example.com").send()?;
let results = su.amazon_search("wireless headphones").marketplace("amazon.de").send()?;

Like reqwest::blocking, it runs its own small runtime, so do not use it inside async code.

Error handling

Every call returns scrapeunblocker::Result<T>. An error response from the API is Error::Api, whose kind says what went wrong:

use scrapeunblocker::{ApiErrorKind, Error};

match su.get_page_source("https://example.com").await {
    Ok(html) => println!("{} bytes", html.len()),
    Err(Error::Api(e)) => match e.kind {
        ApiErrorKind::Blocked => {}                       // 403: blocked on every bypass path (not billed)
        k if k.is_payment_required() => {}                // 402: quota, credit limit, budget or failed payment
        ApiErrorKind::RateLimit => {}                     // 429: over your concurrency limit
        ApiErrorKind::UpstreamOutage => {}                // 503: the target site itself is down
        ApiErrorKind::TargetNotFound => {
            let t = e.target_not_found.as_ref().unwrap();  // 404/410 from the target (billed)
            println!("{} {:?}", t.origin_status, t.html);
        }
        _ => println!("{e}"),
    },
    Err(Error::Timeout(_)) => {}                          // this client gave up locally
    Err(e) => println!("{e}"),
}
ApiErrorKind Status Meaning
InvalidRequest 400 Bad URL, unsupported scheme, or the API key header was not sent
Authentication 401 Key not recognised - typo, stray whitespace, or a rotated key
NoSubscription 401 Key is fine, but the account has no active plan
QuotaExceeded 402 The plan's requests for this period are used up
CreditLimitExceeded 402 Unpaid balance is past the account's credit limit
BudgetExceeded 402 This period's spend reached the monthly budget limit you set in your profile
PaymentFailed 402 A card payment was declined three times
PaymentRequired 402 Any other billing block
Blocked 403 Blocked by bot protection on every path
NotFound 404 What you asked for does not exist - a plugin lookup found nothing
TargetNotFound 404 / 410 The target page itself does not exist; target_not_found has origin_status, html, destination_url (billed)
BrowserTimeout 408 Our browser run timed out before the page was ready
UnsupportedContent 415 The URL serves something other than HTML (get_page_source, get_markdown) or other than an image (get_image)
Validation 422 Missing or wrong-typed parameter; body holds the detail array
StepFailed 422 A browser step failed; step_failed has step_index, action, reason, selector, html
RateLimit 429 Too many concurrent requests
UpstreamOutage 503 The target origin is down
Server 5xx Unexpected server error, including a 504 upstream timeout

Outside Error::Api: Error::Timeout (this client gave up locally), Error::Connection (could not reach the API), Error::MissingApiKey, Error::Decode and Error::Config.

HTTP 429, 502, 503, 504 and network errors are retried automatically with exponential backoff (ClientBuilder::max_retries). A 401 or 402 is never retried - it clears when the key or the billing state changes - and neither is billed.

TLS

The default TLS backend is reqwest's default. To choose another, turn off default features:

scrapeunblocker = { version = "0.20", default-features = false, features = ["rustls"] }  # or "native-tls"

Links

License

MIT