Petrify

Petrify is a command-line tool that downloads a website and produces a static copy suitable for offline browsing or hosting behind a web server. It crawls HTML pages on the target host, extracts linked assets, rewrites URLs to site-root paths, and saves everything to disk.

Petrify is similar in purpose to wget --mirror, with built-in link rewriting, concurrent downloads, optional image conversion to WebP, and discovery of SEO / social metadata images.

What Petrify does

Given a starting URL, Petrify will:

Discover HTML pages on the same host by following links.
Download each page, parse it, and rewrite href, src, style url(...), SEO meta images, and related references to local site-root paths.
Download referenced assets (CSS, JavaScript, images, fonts, video, PDF, and other selected types), including common off-host CDNs and object storage when linked from HTML.
Optionally convert JPEG, PNG, and GIF images to WebP during download (HTML links are updated to match).
Write a directory tree you can open locally or serve as a static site document root.

Petrify does not execute JavaScript or render pages in a browser. It works from the HTML and asset URLs present in the source.

How it works

Petrify runs in four stages:

Scan. Starting from the URL you provide, Petrify downloads each HTML page on the same domain, extracts links and asset references, and builds a queue of pages to process and resources to download. A progress spinner shows scan status.

Report. Before downloading assets, Petrify prints a summary: target URL, output directory, page count, resource count, and active options.

Process pages. Each queued page is downloaded again, parsed, rewritten, and saved under the output directory. Pages are processed concurrently (see --max-concurrent).

Download resources. Assets are grouped by type, downloaded concurrently, and stored under static/. Each URL is downloaded at most once. Failed requests (including 404 responses) are logged with the URL.

Domain scope

During the scan phase, only pages on the same host as the starting URL are followed. For example, if you start at https://example.com/, pages on https://other.com/ linked from the site are not added to the crawl queue.

Asset URLs (images, CSS, JS, fonts, and so on) may be collected from the starting host and from external hosts when they appear in HTML attributes, inline CSS, or SEO metadata. See Limitations for notes on --download-external.

Link rewriting

The HTML parser resolves relative URLs, absolute paths, absolute URLs, and protocol-relative URLs (//cdn.example.com/...). It reads URLs from:

Common attributes (src, href, data-src, poster, and others)
Inline CSS (@import and url(...) in <style> and style="…")
SEO / social metadata (og:image, twitter:image, and related content values)
JSON-LD image-like fields (image, thumbnailUrl, contentUrl, logo, photo)

Rewritten links use site-root paths (for example /static/images/logo.webp and /about/index.html) so nested pages work when the output directory is served as a web server document root. URL fragments such as #home are preserved (/page/index.html#home).

Installation

From crates.io

cargo install petrify

From GitHub Releases

Prebuilt binaries for Linux, macOS, and Windows are attached to each GitHub release.

From source

git clone https://github.com/thanos/petrify.git
cd petrify
cargo install --path .

Requires Rust 1.74 or later.

Quick start

# Download a site to ./petrified_site
petrify https://example.com

# Choose output directory and worker count
petrify https://example.com -o ./my_copy -m 4

# Limit how many pages are scanned and processed (useful for testing)
petrify https://example.com --max-pages 10

# Narrow the asset types (fonts are included by default)
petrify https://example.com --download-only html,css,js,images

Serving the mirror

Point a static file server at the output directory (the document root):

cd petrified_site
python3 -m http.server 8000
# open http://127.0.0.1:8000/

Because links are site-root absolute (/static/...), nested pages such as /artfestival/artist/name/ resolve assets correctly.

Command reference

petrify [OPTIONS] <URL>

Option	Short	Description	Default
`<URL>`		Starting URL to crawl (required)
`--output`	`-o`	Directory for the static copy	`./petrified_site`
`--max-concurrent`	`-m`	Number of concurrent download workers	CPU core count
`--download-only`		Comma-separated list of resource types to download	`js,css,images,video,html,pdf,fonts`
`--max-pages`		Maximum pages to scan and process (`0` = no limit)	`0`
`--depth`	`-d`	Maximum crawl depth	`unlimited`
`--convert-to-webp`		Convert JPEG, PNG, and GIF images to WebP	`true`
`--webp-quality`		WebP quality when lossy compression is used (1 to 100)	`75`
`--webp-lossless`		Use lossless WebP compression instead of quality setting	`false`
`--download-external`		Download resources from external hosts	`true`
`--ignore-robots`	`-i`	Ignore robots.txt restrictions	`true`
`--timeout`		HTTP request timeout in seconds	`270`
`--help`	`-h`	Print help
`--version`	`-V`	Print version

Run petrify --help for the full list.

Resource types

The --download-only flag accepts these values:

Value	File types
`html`	HTML pages (`.html`, `.htm`, extensionless paths treated as pages)
`css`	Stylesheets
`js`	JavaScript
`images`	Images (`.jpg`, `.jpeg`, `.png`, `.gif`, `.webp`, `.svg`, `.ico`, and related)
`video`	Video (`.mp4`, `.webm`, `.ogg`, `.avi`, `.mov`, and related)
`pdf`	PDF documents
`fonts`	Web fonts (`.woff`, `.woff2`, `.ttf`, `.otf`, `.eot`)
`other`	Everything else

fonts is included in the default list. Meta / Open Graph / Twitter image URLs and JSON-LD image fields are discovered and rewritten when images is enabled.

To exclude fonts:

petrify https://example.com --download-only html,css,js,images

Page limit

--max-pages limits how many pages are discovered during the scan and how many are written during processing. It does not cap the number of assets downloaded. Assets referenced by the pages that were scanned are still queued and downloaded according to --download-only.

WebP conversion

When --convert-to-webp is enabled (the default), JPEG, PNG, and GIF images are converted to WebP before being saved. The file extension in the output directory changes to .webp, and HTML references are rewritten to match. Disable conversion to keep original image formats:

petrify https://example.com --convert-to-webp false

Output layout

Petrify writes HTML pages under the output directory, preserving URL structure where possible. Assets are stored under static/ by type.

petrified_site/
├── index.html              # https://example.com/
├── about.html              # https://example.com/about or about.html
├── blog/
│   └── post/
│       └── index.html      # https://example.com/blog/post/
└── static/
    ├── css/
    ├── js/
    ├── images/
    ├── video/
    ├── pdf/
    ├── fonts/
    └── other/

Path mapping examples:

URL	Output file	Rewritten link
`https://example.com/`	`index.html`	`/index.html`
`https://example.com/about`	`about.html`	`/about.html`
`https://example.com/docs/`	`docs/index.html`	`/docs/index.html`
`https://example.com/docs/#intro`	`docs/index.html`	`/docs/index.html#intro`
`https://example.com/static/app.css`	`static/css/app.css`	`/static/css/app.css`
`https://example.com/logo.png`	`static/images/logo.webp` (if conversion is on)	`/static/images/logo.webp`

Examples

Documentation site

petrify https://docs.example.com \
  -o ./docs_offline \
  --download-only html,css,js,images,fonts \
  -m 4

Images as WebP with higher quality

petrify https://example.com \
  -o ./site \
  --convert-to-webp \
  --webp-quality 85

Small test run

petrify https://example.com \
  -o ./test_run \
  --max-pages 5 \
  --max-concurrent 2

Logging

Petrify uses the standard RUST_LOG environment variable (via env_logger). To see warnings, including HTTP errors and 404 responses with the failing URL:

RUST_LOG=petrify=warn petrify https://example.com

For more detail during development:

RUST_LOG=petrify=info,petrify=debug petrify https://example.com

Library

Petrify is also available as a Rust library. The main entry point is petrify::engine::Petrifier:

use petrify::config::Config;
use petrify::engine::Petrifier;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut config = Config::new();
    config.url = "https://example.com".to_string();
    config.output = "./petrified_site".to_string();

    let mut petrifier = Petrifier::new(config).await?;
    petrifier.run().await?;
    Ok(())
}

API documentation is published at docs.rs/petrify.

Development

Build and test

cargo build
cargo test
cargo clippy --all-targets -- -D warnings
cargo fmt --all

Integration tests use the fixture in test_site/ and a local HTTP mock server (wiremock).

Coverage

CI runs tests with tarpaulin and uploads results to Coveralls. To run coverage locally on Linux:

cargo install cargo-tarpaulin
cargo tarpaulin --all-targets --out Html --output-dir coverage

On macOS, cargo-llvm-cov is a practical alternative.

Manual smoke test

./test_petrify.sh

This script builds the release binary and runs petrify against a few public URLs. It requires network access.

Release

See CHANGELOG.md for release notes. Tagging v* on the default branch triggers GitHub Actions to build binaries, create a GitHub release, and publish to crates.io (requires the CRATES_IO_TOKEN repository secret).

# After merging release-ready changes to main:
# 1. Ensure version in Cargo.toml matches the tag (e.g. 0.2.0)
# 2. Commit CHANGELOG + version bump
# 3. Tag and push
git tag -a v0.2.0 -m "Release 0.2.0"
git push origin v0.2.0

Limitations

Petrify produces a static snapshot. It has the following constraints:

No JavaScript rendering. Content loaded or modified only by client-side JavaScript after page load will not appear in the output. Single-page applications may not archive completely.
Same-host crawling. The scan phase only follows links to pages on the same host as the starting URL.
Depth not enforced. --depth is accepted on the command line but is not currently applied during crawling.
External resources. --download-external is accepted but not fully enforced as a hard gate. Off-host assets referenced in HTML may still be queued depending on how they are linked.
robots.txt. --ignore-robots is accepted but robots.txt is not fetched or checked. Use Petrify only on sites you are permitted to copy.
HTTP errors. Failed downloads are logged and skipped. The run continues; the output may contain broken references where a resource could not be fetched.
CSS-linked fonts. Fonts referenced only from downloaded CSS files (not from HTML) are not yet crawled as a second pass.
Scale. Large sites require significant time, bandwidth, and disk space. Use --max-pages to test on a subset first.

Changelog

See CHANGELOG.md for the full history. Latest release: 0.2.0.

License

Licensed under the Apache License, Version 2.0. See LICENSE.

petrify 0.2.0