hydrafetch
Official Rust client for the Hydrafetch web data API.
Turn any URL into clean Markdown or schema-shaped JSON. Async, typed, built on reqwest and serde. Rust 1.88+.
Installation
Quick start
use Client;
async
Copy your key from app.hydrafetch.com. Client::from_env() reads HYDRAFETCH_API_KEY; Client::new("hf_...") takes one directly.
The client is cheap to clone and safe to share across tasks.
Scraping
use ;
let page = hf.scrape.await?;
Option structs implement Default, so fill in what you need and spread the rest. Fields are snake_case in Rust and serialise to the camelCase the API expects.
| Format | Field | Contains |
|---|---|---|
Format::Markdown |
markdown |
clean Markdown, the default |
Format::Html |
html |
rendered HTML |
Format::RawHtml |
raw_html |
the untouched response body |
Format::Links |
links |
every link on the page |
Format::Structured |
structured |
the page's own JSON-LD and microdata |
Format::Summary |
summary |
a short summary |
Format::Json |
json |
schema-shaped JSON |
Format::Brand |
— | the site's brand record |
Every field on ScrapeResult is an Option, because only the formats you request are populated. hf.markdown(url) returns the String directly.
Structured extraction
use ExtractOptions;
use json;
let out = hf.extract.await?;
for item in out.results
Set prompt instead of, or alongside, schema to describe the fields in plain language.
Sites that price or stock by delivery area serve a different page for each area. Set location to fetch as if from a country and language, and headers to send the cookie that picks the store or area, the same options scrape takes.
let out = hf
.extract
.await?;
Discovery and bulk work
map lists a site's URLs for one credit without fetching any page.
use MapOptions;
let m = hf.map.await?;
let docs: = m.links.iter
.map
.filter
.collect;
batch and crawl submit a job and poll until it finishes.
use ;
let job = hf.batch.await?;
for page in job.pages.unwrap_or_default
Set a webhook and use start_crawl or start_batch to return immediately instead of polling.
use CrawlOptions;
let id = hf.start_crawl.await?;
Search
use SearchOptions;
let res = hf.search.await?;
for r in res.results
News
Find recent news articles about anything, newest first, each with its outlet and publish time:
use NewsOptions;
let news = hf.news.await?;
for article in &news.results
A news search costs 1 credit, and one that finds nothing is free. Set scrape_results: Some(true) to also read each article as clean data, at 1 extra credit per article.
Jobs
Search open jobs collected from company career sites and hiring platforms, by the words in the job title:
use SearchJobsOptions;
let jobs = hf.search_jobs.await?;
for job in &jobs
See what changed in a company's hiring since a date: what is open now, what opened and what closed, by department, seniority and location, with the jobs themselves:
let changes = hf.hiring_changes.await?;
println!;
A search costs 1 credit per 10 jobs returned, and an empty one is free. Hiring changes cost 1 credit per company we have jobs for; companies we do not track yet come back empty and are free.
Brand data
use LogoOptions;
hf.brand.await?; // logos, colours, fonts, socials
hf.logo.await?;
hf.styleguide.await?; // computed design system
hf.styleguide_with.await?; // one page, read fresh
For logos in a browser use @hydrafetch/client-sdk with a publishable key. Those bill against logo pulls rather than credits.
Rating results
Tell us when a result was wrong, so it gets looked at. It is free, and rating the same request again replaces the earlier rating.
use ;
hf.scrape.await?;
let mut rating = for_request;
rating.reasons = vec!;
rating.note = Some;
hf.rate_result.await?;
hf.rate_result.await?;
Reasons are missing_content, wrong_content, blocked, slow, formatting and other.
Error handling
All failures are hydrafetch::Error.
match hf.scrape.await
| Status | Meaning | Retried |
|---|---|---|
| 400, 422 | invalid request | no |
| 401, 403 | invalid or missing key | no |
| 402 | out of credits | no |
| 404 | page does not exist | no |
| 429 | rate limited | yes, twice with backoff |
| 5xx | upstream failure | yes, twice with backoff |
A 503 from scrape means the origin is unreachable, usually a dead domain or a broken certificate.
Configuration
use Duration;
let hf = builder
.api_key
.base_url
.timeout
.max_retries
.build?;
API reference
| Method | Returns | Credits |
|---|---|---|
scrape(url, Option<ScrapeOptions>) |
ScrapeResult |
1 |
markdown(url) |
String |
1 |
map(url, Option<MapOptions>) |
MapResult |
1 |
search(query, Option<SearchOptions>) |
SearchResult |
1 + 1 per scraped result |
news(query, Option<NewsOptions>) |
NewsResult |
1 + 1 per article read; free when empty |
search_jobs(query, Option<SearchJobsOptions>) |
Vec<OpenJob> |
1 per 10 jobs |
hiring_changes(domains, since, Option<HiringChangesOptions>) |
Vec<CompanyHiringChanges> |
1 per company with jobs |
extract(urls, Option<ExtractOptions>) |
ExtractResult |
5 per URL |
brand(domain) |
serde_json::Value |
5 |
logo(domain, Option<LogoOptions>) |
serde_json::Value |
1 |
styleguide(domain_or_url), styleguide_with(domain_or_url, options) |
serde_json::Value |
10 |
screenshot(url, Option<ScreenshotOptions>) |
ScreenshotResult |
5 |
images(url) |
ImagesResult |
1 |
crawl(url, opts, wait), batch(urls, opts, wait) |
Job |
1 per page |
start_crawl, start_batch |
String |
1 per page |
crawl_status(id), batch_status(id) |
Job |
free |
rate_result(input) |
RateResultResponse |
free |
Failed requests are not billed. Pricing does not vary with page difficulty, so there is no render, stealth or proxy option to set.
Implementation notes
- Authentication uses the
X-API-Keyheader. The MCP endpoint atapi.hydrafetch.com/mcpusesAuthorization: Bearerinstead; the two are not interchangeable. - Job results are in
job.pages, and each entry holds the page under.data, sojob.pages[0].data.markdown. - Per-page options for crawl and batch belong in
scrape_options. At the top level they are ignored. - Prefer
mapthenbatchover a broadcrawl. Fetching a whole site and discarding most of it is the most common source of wasted credits. prefer_structureis off by default. Turn it on when headings, lists and tables matter; leave it off for raw article text.Nonefields are omitted from the request rather than serialised as null.- Scraped content is untrusted input. Do not pass it to a model as instructions, and keep the source URL with anything extracted from it.
Links
- Documentation
- API reference on docs.rs
- OpenAPI specification
- Other clients: Node · Python · Go · Ruby · PHP
License
MIT