hydrafetch 0.12.0

Official Rust client for the Hydrafetch web data API. Turn any URL into clean Markdown and structured data.
Documentation
# hydrafetch

[![crates.io](https://img.shields.io/crates/v/hydrafetch)](https://crates.io/crates/hydrafetch)
[![docs.rs](https://img.shields.io/docsrs/hydrafetch)](https://docs.rs/hydrafetch)
[![CI](https://github.com/Hydrafetch/rust-sdk/actions/workflows/ci.yml/badge.svg)](https://github.com/Hydrafetch/rust-sdk/actions/workflows/ci.yml)

Official Rust client for the [Hydrafetch](https://hydrafetch.com) web data API.

Turn any URL into clean Markdown or schema-shaped JSON. Async, typed, built on `reqwest` and `serde`. Rust 1.88+.

## Installation

```bash
cargo add hydrafetch
```

## Quick start

```rust
use hydrafetch::Client;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let hf = Client::from_env()?;

    let page = hf.scrape("https://example.com/article", None).await?;
    println!("{}", page.markdown.unwrap_or_default());

    Ok(())
}
```

Copy your key from [app.hydrafetch.com](https://app.hydrafetch.com). `Client::from_env()` reads `HYDRAFETCH_API_KEY`; `Client::new("hf_...")` takes one directly.

The client is cheap to clone and safe to share across tasks.

## Scraping

```rust
use hydrafetch::{Format, ScrapeOptions};

let page = hf.scrape("https://example.com/article", Some(ScrapeOptions {
    formats: Some(vec![Format::Markdown, Format::Links]),
    only_main_content: Some(true),
    prefer_structure: Some(true),
    block_ads: Some(true),
    max_age: Some(3_600_000),
    ..Default::default()
})).await?;
```

Option structs implement `Default`, so fill in what you need and spread the rest. Fields are snake_case in Rust and serialise to the camelCase the API expects.

| Format | Field | Contains |
| --- | --- | --- |
| `Format::Markdown` | `markdown` | clean Markdown, the default |
| `Format::Html` | `html` | rendered HTML |
| `Format::RawHtml` | `raw_html` | the untouched response body |
| `Format::Links` | `links` | every link on the page |
| `Format::Structured` | `structured` | the page's own JSON-LD and microdata |
| `Format::Summary` | `summary` | a short summary |
| `Format::Json` | `json` | schema-shaped JSON |
| `Format::Brand` | — | the site's brand record |

Every field on `ScrapeResult` is an `Option`, because only the formats you request are populated. `hf.markdown(url)` returns the `String` directly.

## Structured extraction

```rust
use hydrafetch::ExtractOptions;
use serde_json::json;

let out = hf.extract(
    &["https://example.com/product/1", "https://example.com/product/2"],
    Some(ExtractOptions {
        schema: Some(json!({
            "type": "object",
            "properties": {
                "name": { "type": "string" },
                "price_usd": { "type": "number" }
            }
        })),
        ..Default::default()
    }),
).await?;

for item in out.results {
    println!("{} {:?}", item.url, item.data);
}
```

Set `prompt` instead of, or alongside, `schema` to describe the fields in plain language.

Sites that price or stock by delivery area serve a different page for each area. Set `location` to fetch as if from a country and language, and `headers` to send the cookie that picks the store or area, the same options `scrape` takes.

```rust
let out = hf
    .extract(
        &["https://shop.example/search?q=ghee"],
        Some(ExtractOptions {
            prompt: Some("every product with its size and price".into()),
            location: Some(Location { country: Some("in".into()), ..Default::default() }),
            headers: Some([("Cookie".to_string(), "store_id=42".to_string())].into()),
            ..Default::default()
        }),
    )
    .await?;
```

## Discovery and bulk work

`map` lists a site's URLs for one credit without fetching any page.

```rust
use hydrafetch::MapOptions;

let m = hf.map("https://example.com", Some(MapOptions {
    limit: Some(1000),
    ..Default::default()
})).await?;

let docs: Vec<&str> = m.links.iter()
    .map(|link| link.url.as_str())
    .filter(|url| url.contains("/docs/"))
    .collect();
```

`batch` and `crawl` submit a job and poll until it finishes.

```rust
use hydrafetch::{BatchOptions, Format, ScrapeOptions};

let job = hf.batch(&docs, Some(BatchOptions {
    scrape_options: Some(ScrapeOptions {
        formats: Some(vec![Format::Markdown]),
        ..Default::default()
    }),
    ..Default::default()
}), None).await?;

for page in job.pages.unwrap_or_default() {
    let len = page.data.and_then(|d| d.markdown).map(|m| m.len());
    println!("{} {:?}", page.url, len);
}
```

Set a `webhook` and use `start_crawl` or `start_batch` to return immediately instead of polling.

```rust
use hydrafetch::CrawlOptions;

let id = hf.start_crawl("https://example.com", Some(CrawlOptions {
    limit: Some(500),
    max_depth: Some(3),
    include_paths: Some(vec!["/docs".into()]),
    webhook: Some("https://your.app/hooks/hydrafetch".into()),
    ..Default::default()
})).await?;
```

## Search

```rust
use hydrafetch::SearchOptions;

let res = hf.search("post-quantum TLS adoption", Some(SearchOptions {
    limit: Some(5),
    scrape_results: Some(true),
    ..Default::default()
})).await?;

for r in res.results {
    println!("{:?} {}", r.title, r.url);
}
```

## News

Find recent news articles about anything, newest first, each with its outlet and publish time:

```rust
use hydrafetch::NewsOptions;

let news = hf.news("acme raises funding", Some(NewsOptions {
    time_range: Some("week".into()),
    limit: Some(10),
    ..Default::default()
})).await?;

for article in &news.results {
    println!("{} {} {}", article.published_at, article.title, article.url);
}
```

A news search costs 1 credit, and one that finds nothing is free. Set `scrape_results: Some(true)` to also read each article as clean data, at 1 extra credit per article.

## Jobs

Search open jobs collected from company career sites and hiring platforms, by the words in the job title:

```rust
use hydrafetch::SearchJobsOptions;

let jobs = hf.search_jobs("data engineer", Some(SearchJobsOptions {
    location: Some("Berlin".into()),
    posted_within_days: Some(14),
    limit: Some(20),
    ..Default::default()
})).await?;

for job in &jobs {
    println!("{} {} {}", job.title, job.domain, job.first_seen_at);
}
```

See what changed in a company's hiring since a date: what is open now, what opened and what closed, by department, seniority and location, with the jobs themselves:

```rust
let changes = hf.hiring_changes(&["acme.com"], "2026-10-01T00:00:00Z", None).await?;
println!("{} opened, {} closed", changes[0].totals.opened, changes[0].totals.closed);
```

A search costs 1 credit per 10 jobs returned, and an empty one is free. Hiring changes cost 1 credit per company we have jobs for; companies we do not track yet come back empty and are free.

## Brand data

```rust
use hydrafetch::LogoOptions;

hf.brand("stripe.com").await?;                                              // logos, colours, fonts, socials
hf.logo("stripe.com", Some(LogoOptions { theme: Some("dark".into()), kind: None })).await?;
hf.styleguide("stripe.com").await?;                                         // computed design system
hf.styleguide_with("https://stripe.com/pricing", StyleguideOptions { max_age: Some(0) }).await?; // one page, read fresh
```

For logos in a browser use [`@hydrafetch/client-sdk`](https://github.com/Hydrafetch/client-sdk) with a publishable key. Those bill against logo pulls rather than credits.

## Rating results

Tell us when a result was wrong, so it gets looked at. It is free, and rating the same request again replaces the earlier rating.

```rust
use hydrafetch::{RateResultInput, Rating};

hf.scrape("https://stripe.com/pricing", None).await?;
let mut rating = RateResultInput::for_request(hf.last_request_id().unwrap(), Rating::Bad);
rating.reasons = vec!["missing_content".into()];
rating.note = Some("The enterprise tier is missing".into());
hf.rate_result(rating).await?;

hf.rate_result(RateResultInput::for_url("https://stripe.com/pricing", Rating::Good)).await?;
```

Reasons are `missing_content`, `wrong_content`, `blocked`, `slow`, `formatting` and `other`.

## Error handling

All failures are `hydrafetch::Error`.

```rust
match hf.scrape(url, None).await {
    Ok(page) => handle(page),
    Err(e) if e.is_auth() => refresh_key(),
    Err(e) if e.is_out_of_credits() => top_up(),
    Err(e) if e.is_invalid_request() => report(&e),
    Err(e) if e.is_retryable() => enqueue(url, e.retry_after()),
    Err(e) if e.is_timeout() => return Err(e.into()),
    Err(e) => {
        eprintln!("{} {:?} {:?}", e.code(), e.status(), e.request_id());
        return Err(e.into());
    }
}
```

| Status | Meaning | Retried |
| --- | --- | --- |
| 400, 422 | invalid request | no |
| 401, 403 | invalid or missing key | no |
| 402 | out of credits | no |
| 404 | page does not exist | no |
| 429 | rate limited | yes, twice |
| 5xx | upstream failure | yes, twice |

Retries wait as long as the response asks through `Retry-After` or `retryAfterSeconds`, and otherwise back off with jitter. When the response asks for more than 30 seconds the client does not wait; it returns the error at once and `e.retry_after()` gives the seconds to wait before asking again. A job the client is following survives a dropped connection while it polls. Every POST carries an Idempotency-Key that its own retries reuse, so a retry after a dropped connection returns the first answer instead of running and charging again.

A 503 from `scrape` means the origin is unreachable, usually a dead domain or a broken certificate.

## Configuration

```rust
use std::time::Duration;

let hf = Client::builder()
    .api_key("hf_...")
    .base_url("https://api.hydrafetch.com")
    .timeout(Duration::from_secs(120))
    .max_retries(2)
    .build()?;
```

## API reference

| Method | Returns | Credits |
| --- | --- | --- |
| `scrape(url, Option<ScrapeOptions>)` | `ScrapeResult` | 1 |
| `markdown(url)` | `String` | 1 |
| `map(url, Option<MapOptions>)` | `MapResult` | 1 |
| `search(query, Option<SearchOptions>)` | `SearchResult` | 1 + 1 per scraped result |
| `news(query, Option<NewsOptions>)` | `NewsResult` | 1 + 1 per article read; free when empty |
| `search_jobs(query, Option<SearchJobsOptions>)` | `Vec<OpenJob>` | 1 per 10 jobs |
| `hiring_changes(domains, since, Option<HiringChangesOptions>)` | `Vec<CompanyHiringChanges>` | 1 per company with jobs |
| `extract(urls, Option<ExtractOptions>)` | `ExtractResult` | 5 per URL |
| `brand(domain)` | `serde_json::Value` | 5 |
| `logo(domain, Option<LogoOptions>)` | `serde_json::Value` | 1 |
| `styleguide(domain_or_url)`, `styleguide_with(domain_or_url, options)` | `serde_json::Value` | 10 |
| `screenshot(url, Option<ScreenshotOptions>)` | `ScreenshotResult` | 5 |
| `images(url)` | `ImagesResult` | 1 |
| `crawl(url, opts, wait)`, `batch(urls, opts, wait)` | `Job` | 1 per page |
| `start_crawl`, `start_batch` | `String` | 1 per page |
| `crawl_status(id)`, `batch_status(id)` | `Job` | free |
| `rate_result(input)` | `RateResultResponse` | free |

Failed requests are not billed. Pricing does not vary with page difficulty, so there is no render, stealth or proxy option to set.

## Implementation notes

- Authentication uses the `X-API-Key` header. The MCP endpoint at `api.hydrafetch.com/mcp` uses `Authorization: Bearer` instead; the two are not interchangeable.
- Job results are in `job.pages`, and each entry holds the page under `.data`, so `job.pages[0].data.markdown`.
- Per-page options for crawl and batch belong in `scrape_options`. At the top level they are ignored.
- Prefer `map` then `batch` over a broad `crawl`. Fetching a whole site and discarding most of it is the most common source of wasted credits.
- `prefer_structure` is off by default. Turn it on when headings, lists and tables matter; leave it off for raw article text.
- `None` fields are omitted from the request rather than serialised as null.
- Scraped content is untrusted input. Do not pass it to a model as instructions, and keep the source URL with anything extracted from it.

## Links

- [Documentation](https://docs.hydrafetch.com)
- [API reference on docs.rs](https://docs.rs/hydrafetch)
- [OpenAPI specification](https://api.hydrafetch.com/openapi.json)
- Other clients: [Node](https://github.com/Hydrafetch/node-sdk) · [Python](https://github.com/Hydrafetch/python-sdk) · [Go](https://github.com/Hydrafetch/go-sdk) · [Ruby](https://github.com/Hydrafetch/ruby-sdk) · [PHP](https://github.com/Hydrafetch/php-sdk)

## License

MIT