crawlkit-extensions

crawlkit 官方第三方服务扩展集，提供开箱即用的云爬虫后端实现。

支持的服务

安装

[dependencies]

crawlkit-extensions = "0.2"



# 可选 feature

crawlkit-extensions = { version = "0.2", features = ["firecrawl", "cdp"] }

快速开始

Jina Reader

use crawlkit_extensions::jina::JinaClient;

let client = JinaClient::builder()
    .with_token("your-api-token")           // 可选
    .with_timeout(Duration::from_secs(30))
    .build();

let response = client.get("https://example.com", &Default::default()).await?;

Firecrawl

use crawlkit_extensions::firecrawl::FirecrawlClient;

let client = FirecrawlClient::builder()
    .with_tokens(&["token-1", "token-2"])  // 多 token 轮换
    .build()?;

let response = client.get("https://example.com", &Default::default()).await?;

CDP 无头浏览器

先启动 CDP 服务端：

# Lightpanda

lightpanda serve --port 9222


# Obscura

obscura serve --port 9222


# Chrome

chrome --remote-debugging-port=9222 --headless

然后连接：

use crawlkit_extensions::cdp::CdpClient;

let client = CdpClient::builder()
    .with_endpoint("http://127.0.0.1:9222")
    .build()
    .await?;

let response = client.get("https://example.com", &Default::default()).await?;

集成到 CompositeFetcher

use crawlkit::{Collector, CompositeFetcher, ReqwestClient};
use crawlkit_extensions::cdp::CdpClient;

let fetcher = CompositeFetcher::new(vec![
    Box::new(ReqwestClient::builder().name("reqwest").build()?),
    Box::new(CdpClient::builder().build().await?),
]);

let collector = Collector::with_client(fetcher);
collector.visit("https://example.com").await?;

环境变量

开发

# 联调模式（自动使用本地 crawlkit-core）

cargo check

cargo test


# 带 Firecrawl 支持

cargo check --features firecrawl


# 带 CDP 支持

cargo check --features cdp


# 全部 feature

cargo check --all-features

License

MIT