Crate webscrapingapi

WebScrapingAPI

WebScrapingApi is an API that allows scraping websites while using rotating proxies to prevent bans. This SDK for Rust makes the usage of the API easier to implement in any project you have.

Using the SDK it’s quite easy. An example of a GET call to the API is the following:

let mut query_builder = QueryBuilder::new();

query_builder.url("http://httpbin.org/headers");
query_builder.render_js("1");

let mut headers: HashMap<String, String> = HashMap::new();
headers.insert("Wsa-test".to_string(), "abcd".to_string());

query_builder.headers(headers);

let html = wsa.get(query_builder).await?.text().await?;

println!("{}", html);

As sometimes WebScrapingAPI might update their parameters, it is useful to have a raw_get method as well. This is useful in order to be able to add any new parameters implemented by WebScrapingAPI. Here’s how to do a raw_get request.

let mut params: HashMap<&str, &str> = HashMap::new();
params.insert("url", "http://httpbin.org/headers");
 
let mut headers: HashMap<String, String> = HashMap::new();
headers.insert("Wsa-test".to_string(), "abcd".to_string());
 
let html = wsa.raw_get(params, headers).await?.text().await?;

println!("{}", html);

Also for post requests you can just use raw_post and post. For these methods you will need to add the body (HashMap<&str,&str>). To add the body to the query builder use query_builder.body(body); where body is a HashMap<&str, &str> with your body To add the body to a raw_post request use wsa.raw_post(params, headers, body)

Structs

QueryBuilder

The query builder struct that contains the params, headers and body of a request

WebScrapingAPI

The WebScrapingAPI client that makes the requests