Crate webscrape_ai 

Official Rust SDK for the webscrape.ai API.

Covers the public API-key surface: Client::scrape, Client::smartscraper, and the SmartBrowse namespace (dispatch a recipe run, poll it, wait for completion, read usage). See https://webscrape.ai/docs for API docs.

Quick start

use webscrape_ai::{Client, ScrapeRequest};

// Reads WEBSCRAPE_API_KEY from the environment.
let client = Client::from_env()?;

let resp = client
    .scrape(ScrapeRequest::new("https://example.com").clean(true))
    .await?;

println!("credits used: {:?}", resp.credits_used);
println!("markdown: {:?}", resp.data.html);

Structured extraction

use webscrape_ai::{Client, SmartScraperRequest};
use serde_json::json;

let client = Client::new("wsg_live_...")?;
let resp = client
    .smartscraper(
        SmartScraperRequest::new(
            "https://news.ycombinator.com",
            "Extract the front-page stories with title, url, and score.",
        )
        .output_schema(json!({
            "type": "object",
            "properties": { "stories": { "type": "array" } }
        })),
    )
    .await?;

// `data.result` is a serde_json::Value; decode it into your own type.
let value = resp.data.result;
println!("{value:?}");

Error handling

Every method returns Result<T>. Failures surface as Error, which distinguishes API errors (Error::Api carrying an ApiError you branch on via ApiError::kind) from transport, timeout, and wait-helper failures.

Modules

blocking

Synchronous client, enabled by the blocking cargo feature. Mirrors the async Client surface, wrapping reqwest::blocking and using std::thread::sleep for backoff and the wait helper.

Structs

ApiError

A structured API error decoded from the response envelope.

Client

Async webscrape.ai API client. Cheap to Clone (reference-counted).

ClientBuilder

Builder for Client.

LinkInfo

An outbound link from ScrapeData::links.

PageMetadata

Page metadata from ScrapeData::metadata (HTML only; null for PDFs).

Response

A successful (completed/queued) response envelope with typed data.

ScrapeData

data for a successful Client::scrape call.

ScrapeRequest

Request body for Client::scrape.

SmartBrowse

The SmartBrowse method namespace, obtained via Client::smartbrowse.

SmartBrowseDispatch

data returned when dispatching a SmartBrowse run (HTTP 202 queued).

SmartBrowseLastRun

Summary of the account’s most recent run, from SmartBrowseUsage.

SmartBrowseRun

data returned when polling a SmartBrowse run.

SmartBrowseUsage

data for SmartBrowse::usage — plan caps and rolling-30-day usage.

SmartScraperData

data for a successful Client::smartscraper call.

SmartScraperRequest

Request body for Client::smartscraper.

WaitOptions

Options for SmartBrowse::wait_for_run and SmartBrowse::run_and_wait.

Enums

DetailLevel

How exhaustively SmartScraperRequest should populate the result.

Error

Top-level error returned by every SDK method.

ErrorCode

Stable API error code. Unknown codes decode to ErrorCode::Unknown with the raw string preserved.

PageComplexity

Page-complexity hint for SmartScraperRequest.

ParseMode

Cleaner mode used when clean: true.

RunStatus

SmartBrowse run lifecycle state. Unknown values decode to RunStatus::Unknown rather than failing.

Constants

API_KEY_ENV

Environment variable consulted for the API key when none is passed explicitly.

DEFAULT_BASE_URL

Default production base URL (https://api.webscrape.ai/v1).

VERSION

SDK version, sourced from CARGO_PKG_VERSION and used in the User-Agent.

Type Aliases

Result

Convenience alias for Result<T, webscrape_ai::Error>.