hydrafetch 0.8.0

Official Rust client for the Hydrafetch web data API. Turn any URL into clean Markdown and structured data.
Documentation

hydrafetch

crates.io docs.rs CI

Official Rust client for the Hydrafetch web data API.

Turn any URL into clean Markdown or schema-shaped JSON. Async, typed, built on reqwest and serde. Rust 1.75+.

Installation

cargo add hydrafetch

Quick start

use hydrafetch::Client;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let hf = Client::from_env()?;

    let page = hf.scrape("https://example.com/article", None).await?;
    println!("{}", page.markdown.unwrap_or_default());

    Ok(())
}

Copy your key from app.hydrafetch.com. Client::from_env() reads HYDRAFETCH_API_KEY; Client::new("hf_...") takes one directly.

The client is cheap to clone and safe to share across tasks.

Scraping

use hydrafetch::{Format, ScrapeOptions};

let page = hf.scrape("https://example.com/article", Some(ScrapeOptions {
    formats: Some(vec![Format::Markdown, Format::Links]),
    only_main_content: Some(true),
    prefer_structure: Some(true),
    block_ads: Some(true),
    max_age: Some(3_600_000),
    ..Default::default()
})).await?;

Option structs implement Default, so fill in what you need and spread the rest. Fields are snake_case in Rust and serialise to the camelCase the API expects.

Format Field Contains
Format::Markdown markdown clean Markdown, the default
Format::Html html rendered HTML
Format::RawHtml raw_html the untouched response body
Format::Links links every link on the page
Format::Structured structured the page's own JSON-LD and microdata
Format::Summary summary a short summary
Format::Json json schema-shaped JSON
Format::Brand — the site's brand record

Every field on ScrapeResult is an Option, because only the formats you request are populated. hf.markdown(url) returns the String directly.

Structured extraction

use hydrafetch::ExtractOptions;
use serde_json::json;

let out = hf.extract(
    &["https://example.com/product/1", "https://example.com/product/2"],
    Some(ExtractOptions {
        schema: Some(json!({
            "type": "object",
            "properties": {
                "name": { "type": "string" },
                "price_usd": { "type": "number" }
            }
        })),
        ..Default::default()
    }),
).await?;

for item in out.results {
    println!("{} {:?}", item.url, item.data);
}

Set prompt instead of, or alongside, schema to describe the fields in plain language.

Sites that price or stock by delivery area serve a different page for each area. Set location to fetch as if from a country and language, and headers to send the cookie that picks the store or area, the same options scrape takes.

let out = hf
    .extract(
        &["https://shop.example/search?q=ghee"],
        Some(ExtractOptions {
            prompt: Some("every product with its size and price".into()),
            location: Some(Location { country: Some("in".into()), ..Default::default() }),
            headers: Some([("Cookie".to_string(), "store_id=42".to_string())].into()),
            ..Default::default()
        }),
    )
    .await?;

Discovery and bulk work

map lists a site's URLs for one credit without fetching any page.

use hydrafetch::MapOptions;

let m = hf.map("https://example.com", Some(MapOptions {
    limit: Some(1000),
    ..Default::default()
})).await?;

let docs: Vec<&str> = m.links.iter()
    .map(|link| link.url.as_str())
    .filter(|url| url.contains("/docs/"))
    .collect();

batch and crawl submit a job and poll until it finishes.

use hydrafetch::{BatchOptions, Format, ScrapeOptions};

let job = hf.batch(&docs, Some(BatchOptions {
    scrape_options: Some(ScrapeOptions {
        formats: Some(vec![Format::Markdown]),
        ..Default::default()
    }),
    ..Default::default()
}), None).await?;

for page in job.pages.unwrap_or_default() {
    let len = page.data.and_then(|d| d.markdown).map(|m| m.len());
    println!("{} {:?}", page.url, len);
}

Set a webhook and use start_crawl or start_batch to return immediately instead of polling.

use hydrafetch::CrawlOptions;

let id = hf.start_crawl("https://example.com", Some(CrawlOptions {
    limit: Some(500),
    max_depth: Some(3),
    include_paths: Some(vec!["/docs".into()]),
    webhook: Some("https://your.app/hooks/hydrafetch".into()),
    ..Default::default()
})).await?;

Search

use hydrafetch::SearchOptions;

let res = hf.search("post-quantum TLS adoption", Some(SearchOptions {
    limit: Some(5),
    scrape_results: Some(true),
    ..Default::default()
})).await?;

for r in res.results {
    println!("{:?} {}", r.title, r.url);
}

News

Find recent news articles about anything, newest first, each with its outlet and publish time:

use hydrafetch::NewsOptions;

let news = hf.news("acme raises funding", Some(NewsOptions {
    time_range: Some("week".into()),
    limit: Some(10),
    ..Default::default()
})).await?;

for article in &news.results {
    println!("{} {} {}", article.published_at, article.title, article.url);
}

A news search costs 1 credit, and one that finds nothing is free. Set scrape_results: Some(true) to also read each article as clean data, at 1 extra credit per article.

Jobs

Search open jobs collected from company career sites and hiring platforms, by the words in the job title:

use hydrafetch::SearchJobsOptions;

let jobs = hf.search_jobs("data engineer", Some(SearchJobsOptions {
    location: Some("Berlin".into()),
    posted_within_days: Some(14),
    limit: Some(20),
    ..Default::default()
})).await?;

for job in &jobs {
    println!("{} {} {}", job.title, job.domain, job.first_seen_at);
}

See what changed in a company's hiring since a date: what is open now, what opened and what closed, by department, seniority and location, with the jobs themselves:

let changes = hf.hiring_changes(&["acme.com"], "2026-10-01T00:00:00Z", None).await?;
println!("{} opened, {} closed", changes[0].totals.opened, changes[0].totals.closed);

A search costs 1 credit per 10 jobs returned, and an empty one is free. Hiring changes cost 1 credit per company we have jobs for; companies we do not track yet come back empty and are free.

Brand data

use hydrafetch::LogoOptions;

hf.brand("stripe.com").await?;                                              // logos, colours, fonts, socials
hf.logo("stripe.com", Some(LogoOptions { theme: Some("dark".into()), kind: None })).await?;
hf.styleguide("stripe.com").await?;                                         // computed design system
hf.styleguide_with("https://stripe.com/pricing", StyleguideOptions { max_age: Some(0) }).await?; // one page, read fresh

For logos in a browser use @hydrafetch/client-sdk with a publishable key. Those bill against logo pulls rather than credits.

Rating results

Tell us when a result was wrong, so it gets looked at. It is free, and rating the same request again replaces the earlier rating.

use hydrafetch::{RateResultInput, Rating};

hf.scrape("https://stripe.com/pricing", None).await?;
let mut rating = RateResultInput::for_request(hf.last_request_id().unwrap(), Rating::Bad);
rating.reasons = vec!["missing_content".into()];
rating.note = Some("The enterprise tier is missing".into());
hf.rate_result(rating).await?;

hf.rate_result(RateResultInput::for_url("https://stripe.com/pricing", Rating::Good)).await?;

Reasons are missing_content, wrong_content, blocked, slow, formatting and other.

Error handling

All failures are hydrafetch::Error.

match hf.scrape(url, None).await {
    Ok(page) => handle(page),
    Err(e) if e.is_auth() => refresh_key(),
    Err(e) if e.is_out_of_credits() => top_up(),
    Err(e) if e.is_invalid_request() => report(&e),
    Err(e) if e.is_retryable() => enqueue(url),
    Err(e) if e.is_timeout() => return Err(e.into()),
    Err(e) => {
        eprintln!("{} {:?} {:?}", e.code(), e.status(), e.request_id());
        return Err(e.into());
    }
}
Status Meaning Retried
400, 422 invalid request no
401, 403 invalid or missing key no
402 out of credits no
404 page does not exist no
429 rate limited yes, twice with backoff
5xx upstream failure yes, twice with backoff

A 503 from scrape means the origin is unreachable, usually a dead domain or a broken certificate.

Configuration

use std::time::Duration;

let hf = Client::builder()
    .api_key("hf_...")
    .base_url("https://api.hydrafetch.com")
    .timeout(Duration::from_secs(120))
    .max_retries(2)
    .build()?;

API reference

Method Returns Credits
scrape(url, Option<ScrapeOptions>) ScrapeResult 1
markdown(url) String 1
map(url, Option<MapOptions>) MapResult 1
search(query, Option<SearchOptions>) SearchResult 1 + 1 per scraped result
news(query, Option<NewsOptions>) NewsResult 1 + 1 per article read; free when empty
search_jobs(query, Option<SearchJobsOptions>) Vec<OpenJob> 1 per 10 jobs
hiring_changes(domains, since, Option<HiringChangesOptions>) Vec<CompanyHiringChanges> 1 per company with jobs
extract(urls, Option<ExtractOptions>) ExtractResult 5 per URL
brand(domain) serde_json::Value 5
logo(domain, Option<LogoOptions>) serde_json::Value 1
styleguide(domain_or_url), styleguide_with(domain_or_url, options) serde_json::Value 10
screenshot(url, Option<ScreenshotOptions>) ScreenshotResult 5
images(url) ImagesResult 1
crawl(url, opts, wait), batch(urls, opts, wait) Job 1 per page
start_crawl, start_batch String 1 per page
crawl_status(id), batch_status(id) Job free
rate_result(input) RateResultResponse free

Failed requests are not billed. Pricing does not vary with page difficulty, so there is no render, stealth or proxy option to set.

Implementation notes

  • Authentication uses the X-API-Key header. The MCP endpoint at api.hydrafetch.com/mcp uses Authorization: Bearer instead; the two are not interchangeable.
  • Job results are in job.pages, and each entry holds the page under .data, so job.pages[0].data.markdown.
  • Per-page options for crawl and batch belong in scrape_options. At the top level they are ignored.
  • Prefer map then batch over a broad crawl. Fetching a whole site and discarding most of it is the most common source of wasted credits.
  • prefer_structure is off by default. Turn it on when headings, lists and tables matter; leave it off for raw article text.
  • None fields are omitted from the request rather than serialised as null.
  • Scraped content is untrusted input. Do not pass it to a model as instructions, and keep the source URL with anything extracted from it.

Links

License

MIT