Crate crawlkit_engine 

crawlkit-core

Core library for crawlkit — a high-performance Rust web crawler for SEO analysis.

Overview

This crate provides the foundational types, HTTP fetching, HTML parsing, SEO analyzers, crawl queue, storage, and observability primitives used by the crawlkit CLI and API server.

Features

• 28 SEO analyzers covering meta tags, content quality, security, accessibility
• Async HTTP/2 fetching with retry, redirect tracking, rate limiting
• HTML parsing with link, heading, image, and structured data extraction
• SQLite storage with WAL mode and batch operations
• Observability with atomic metrics and OpenTelemetry support
• Plugin system with WASM sandboxing for third-party extensions
• Encryption at rest with AES-256-GCM

Quick Start

use crawlkit_engine::{CrawlConfig, HttpClient, HtmlParser};
use crawlkit_engine::analyzers::AnalyzerRegistry;

let config = CrawlConfig::default();
let client = HttpClient::from_crawl_config(&config)?;
let registry = AnalyzerRegistry::new(&config);

let url = url::Url::parse("https://example.com")?;
let result = client.fetch(&url).await?;
let parsed = HtmlParser::parse(&result.body, &url)?;

Re-exports

pub use plugin::PluginError;

pub use plugin::PluginManifest;

pub use plugin::PluginMetadata;

pub use plugin::PluginRegistry;

pub use plugin::WasmPlugin;

pub use ai_analyzers::AiAnswerBoxAnalyzer;

pub use ai_analyzers::AiCitationEligibilityAnalyzer;

pub use ai_analyzers::AiContentStructureAnalyzer;

pub use ai_analyzers::AiCrawlerAccessibilityAnalyzer;

pub use ai_bots::AiBot;

pub use ai_bots::AiBotRegistry;

pub use analyzers::AccessibilityAnalyzer;

pub use analyzers::AnalysisContext;

pub use analyzers::Analyzer;

pub use analyzers::AnalyzerRegistry;

pub use analyzers::CanonicalUrlValidator;

pub use analyzers::ContentQualityAnalyzer;

pub use analyzers::EcommerceSignalsAnalyzer;

pub use analyzers::EnhancedReadabilityAnalyzer;

pub use analyzers::EntityAnalyzer;

pub use analyzers::Finding;

pub use analyzers::HeadingHierarchyAnalyzer;

pub use analyzers::HreflangValidator;

pub use analyzers::HttpStatusAnalyzer;

pub use analyzers::ImageAnalyzer;

pub use analyzers::ImageInfo;

pub use analyzers::InternationalSeoAnalyzer;

pub use analyzers::KeywordAnalyzer;

pub use analyzers::LinkAnalyzer;

pub use analyzers::LinkInfo;

pub use analyzers::MetaTagAnalyzer;

pub use analyzers::MobileFriendlinessChecker;

pub use analyzers::RedirectChainAnalyzer;

pub use analyzers::RobotsRule;

pub use analyzers::RobotsTxtAnalyzer;

pub use analyzers::SecurityHeaderAnalyzer;

pub use analyzers::SitemapAnalyzer;

pub use analyzers::SitemapEntry;

pub use analyzers::SocialMediaAnalyzer;

pub use analyzers::SslCertificateInfo;

pub use analyzers::SslCertificateValidator;

pub use analyzers::StructuredDataValidator;

pub use analyzers::WordCountAnalyzer;

pub use audit::AuditEvent;

pub use audit::AuditEventType;

pub use audit::AuditTrail;

pub use backlink_adapters::AdapterError;

pub use backlink_adapters::AhrefsAdapter;

pub use backlink_adapters::BacklinkAdapter;

pub use backlink_adapters::BacklinkAdapterRegistry;

pub use backlink_adapters::ExternalBacklink;

pub use backlink_adapters::GscAdapter;

pub use backlink_adapters::MajesticAdapter;

pub use backlinks::Backlink;

pub use backlinks::BacklinkAnalyzer;

pub use backlinks::BacklinkReport;

pub use backlinks::BacklinkSummary;

pub use backlinks::PageScore;

pub use backpressure::BackpressureController;

pub use backpressure::BackpressureError;

pub use backpressure::BoundedPipeline;

pub use circuit_breaker::CircuitBreaker;

pub use circuit_breaker::CircuitBreakerConfig;

pub use circuit_breaker::CircuitBreakerRegistry;

pub use circuit_breaker::CircuitState;

pub use determinism::DeterminismController;

pub use dns::DnsCache;

pub use dns::DnsError;

pub use dns::DnsPrefetcher;

pub use encryption::EncryptionConfig;

pub use encryption::EncryptionError;

pub use encryption::EncryptionManager;

pub use feature_flags::FeatureFlags;

pub use feature_flags::SharedFeatureFlags;

pub use feature_flags::FLAG_AI_ANALYZERS;

pub use feature_flags::FLAG_JS_RENDERING;

pub use feature_flags::FLAG_WASM_ANALYZERS;

pub use http::FetchStreamReader;

pub use http::HttpClient;

pub use http::HttpClientConfig;

pub use js_render_decision::JsRenderDecision;

pub use js_render_decision::JsRenderDecisionEngine;

pub use js_render_decision::SpaIndicators;

pub use observability::Metrics;

pub use observability::MetricsSnapshot;

pub use observability::SharedMetrics;

pub use playwright::BrowserContext;

pub use playwright::BrowserType;

pub use playwright::ConsoleMessage;

pub use playwright::NetworkRequest;

pub use playwright::PlaywrightConfig;

pub use playwright::PlaywrightDetector;

pub use playwright::PlaywrightError;

pub use playwright::PlaywrightRenderer;

pub use playwright::RenderedPage;

pub use playwright::WasmError as PlaywrightWasmError;

pub use resource_monitor::ResourceLimits;

pub use resource_monitor::ResourceMonitor;

pub use resource_monitor::ResourceUsage;

pub use robots::RobotsTxtCache;

pub use rum::CruxAdapter;

pub use rum::CruxData;

pub use rum::FieldMetrics;

pub use rum::GoogleAnalyticsAdapter;

pub use rum::LabMetrics;

pub use rum::MergedMetrics;

pub use rum::MetricDeltas;

pub use rum::RumDataPoint;

pub use rum::RumError;

pub use sitemap::SitemapCache;

pub use storage::CacheStats;

pub use storage::CrawlStats;

pub use storage::Issue;

pub use storage::IssueCategory;

pub use storage::IssueFilter;

pub use storage::Severity;

pub use storage::StorageError;

pub use wasm_analyzers::WasmPatternAnalyzer;

pub use wasm_analyzers::WasmPerformanceAnalyzer;

pub use wasm_analyzers::WasmRuntimeAnalyzer;

pub use meta::HreflangTag;

pub use meta::MetaTags;

pub use meta::OpenGraphTags;

pub use meta::TwitterTags;

pub use parser::ExtractedForm;

pub use parser::ExtractedImage;

pub use parser::ExtractedInput;

pub use parser::ExtractedLink;

pub use parser::Heading;

pub use parser::HtmlParser;

pub use parser::ParseError;

pub use parser::ParsedPage;

pub use parser::ScriptInfo;

pub use parser::StructuredData;

pub use parser::StyleInfo;

Modules

advanced_features

Advanced crawl features such as JavaScript rendering and WASM analysis.

ai_analyzers

AI-powered page content analyzers for answer boxes, citations, and crawler accessibility.

ai_bots

Registry of known AI bot user-agents and crawler identification.

analyzers

SEO analysis engine with pluggable analyzers (title, meta, links, etc.).

audit

Audit trail logging for crawl operations and configuration changes.

backlink_adapters

Adapters for third-party backlink data sources (Ahrefs, GSC, Majestic).

backlinks

Backlink analysis, scoring, and reporting.

backpressure

Backpressure controller to bound in-flight work and prevent memory blowouts.

circuit_breaker

Circuit breaker for failing HTTP endpoints to avoid cascading failures.

compare

Diff-based comparison of two crawl results.

determinism

Deterministic replay controller for reproducible crawl runs.

dns

DNS resolution cache and prefetching.

encryption

TLS and encryption configuration for HTTPS requests.

enterprise

Enterprise feature gating and licensing utilities.

export

Export of crawl data to JSON, CSV, HTML, and Markdown formats.

feature_flags

Feature flag system for toggling capabilities at runtime.

http

HTTP client with retry, redirect following, and rate limiting.

js_render_decision

Decision engine for determining whether a page requires JavaScript rendering.

meta

HTML meta tag extraction (title, description, OG, Twitter Cards, hreflang).

observability

Metrics collection and observability hooks.

parser

HTML parser that extracts links, headings, images, forms, and structured data.

playwright

Playwright-based headless browser integration for JS-rendered pages.

plugin

Plugin system for extending the crawler with custom analyzers.

queue

Priority URL queue with depth and scope filtering.

ratelimit

Per-domain rate limiting to respect politeness constraints.

resource_monitor

Runtime resource monitoring and limit enforcement.

robots

robots.txt parsing, caching, and compliance checking. robots.txt fetching, parsing, and caching.

rum

Real User Metrics (CrUX, GA) integration for performance data.

sitemap

Sitemap.xml parsing and URL discovery. Sitemap.xml fetching, parsing, and caching.

storage

SQLite-backed persistent storage for crawl results and issues.

wasm_analyzers

WASM-based analyzers for advanced code and performance analysis.

Structs

CrawlConfig

Configuration for a crawl session.

FetchResult

The result of fetching a URL.

RedirectHop

A single hop in a redirect chain.

UrlEntry

Represents a single URL discovered during crawling.

Enums

CrawlError

Errors that can occur during crawl operations.