Trait CrawlSink 

pub trait CrawlSink: Send + Sync {
    // Required methods
    fn page(&self, doc: Document) -> BoxFuture<'_, ()>;
    fn error<'a>(&'a self, url: &'a str, err: ApiError) -> BoxFuture<'a, ()>;
    fn total(&self, total: usize) -> BoxFuture<'_, ()>;
    fn cancelled(&self) -> BoxFuture<'_, bool>;

    // Provided methods
    fn reserve_page<'a>(
        &'a self,
        _url: &'a str,
    ) -> BoxFuture<'a, Result<(), ApiError>> { ... }
    fn settle_page<'a>(&'a self, _url: &'a str) -> BoxFuture<'a, ()> { ... }
    fn release_page<'a>(&'a self, _url: &'a str) -> BoxFuture<'a, ()> { ... }
    fn release_pending(&self) -> BoxFuture<'_, ()> { ... }
    fn resume(&self) -> Option<CrawlResume> { ... }
    fn tracks_frontier(&self) -> bool { ... }
    fn enqueued(&self, _urls: Vec<(String, usize)>) -> BoxFuture<'_, ()> { ... }
    fn summary(&self, _summary: CrawlSummary) -> BoxFuture<'_, ()> { ... }
}

整站爬取的进度回报，由 API 层的任务管理实现。

Required Methods

fn page(&self, doc: Document) -> BoxFuture<'_, ()>

fn error<'a>(&'a self, url: &'a str, err: ApiError) -> BoxFuture<'a, ()>

fn total(&self, total: usize) -> BoxFuture<'_, ()>

已经确定要抓的页面总数（随着发现新链接增长）。

fn cancelled(&self) -> BoxFuture<'_, bool>

Provided Methods

fn reserve_page<'a>( &'a self, _url: &'a str, ) -> BoxFuture<'a, Result<(), ApiError>>

发出一次页面抓取之前调用：原子预留这一页的配额。返回 Err 时引擎必须停止调度、 不得再发请求（重试同一 URL 会再次调用，实现要按 URL 幂等）。

fn settle_page<'a>(&'a self, _url: &'a str) -> BoxFuture<'a, ()>

这一页已经成功抓回并会交付：结算预留。之后引擎才调用 CrawlSink::page。

fn release_page<'a>(&'a self, _url: &'a str) -> BoxFuture<'a, ()>

这一页最终失败（不再重试）：退还预留。

fn release_pending(&self) -> BoxFuture<'_, ()>

本次爬取结束：退还未结算的预留（取消、上限、客户端断开都走这里）。

fn resume(&self) -> Option<CrawlResume>

断点续爬：任务是从中断处恢复的时候返回上次的进度，引擎据此重建队列和去重表， 不再抓已经完成的页面。新任务返回 None。

fn tracks_frontier(&self) -> bool

是否需要 CrawlSink::enqueued 回报（持久化的任务需要）。不需要时引擎不必收集。

fn enqueued(&self, _urls: Vec<(String, usize)>) -> BoxFuture<'_, ()>

新进入队列的 URL 和发现深度，按入队顺序。只在 CrawlSink::tracks_frontier 为真时调用。

fn summary(&self, _summary: CrawlSummary) -> BoxFuture<'_, ()>

爬取结束时的说明（停止原因等）；只有带查询的爬取会调用。

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors