crawlkit-engine 2.0.0

High-performance Rust web crawler and SEO analysis toolkit with 28 analyzers, WASM plugin system, and enterprise features

Coverage
82.52%
1048 out of 1270 items documented25 out of 560 items with examples
Size
Source code size: 995.63 kB This is the summed size of all the files inside the crates.io package for this release.
Documentation size: 22.3 MB This is the summed size of all files generated by rustdoc for all configured targets
Ø build duration
this release: 1m 19s Average build duration of successful builds.
all releases: 1m 13s Average build duration of successful builds in releases after 2024-10-23.
Links
Homepage
WyattAu/crawlkit
0 0 0
crates.io
Dependencies
Versions
- 2.0.0 (2026-07-24)
- 1.0.0 (2026-07-24)
Owners

crawlkit-engine

Core library for crawlkit — a high-performance Rust web crawler for SEO analysis.

Features

28 SEO analyzers (meta tags, content quality, security, accessibility, AI)
Async HTTP/2 fetching with retry, redirect tracking, rate limiting
HTML parsing with link, heading, image, and structured data extraction
SQLite storage with WAL mode and batch operations
WASM plugin system for third-party extensions
AES-256-GCM encryption at rest
Deterministic crawls with seed-based reproducibility

Usage

Add to your Cargo.toml:

[dependencies]
crawlkit-engine = "1.0.0"

Documentation