Crate spider_pipeline 

spider-pipeline

Built-in pipeline implementations for the spider-lib framework.

Processes, filters, transforms, and stores scraped data.

Example

use spider_pipeline::json::JsonPipeline;
use spider_pipeline::console::ConsolePipeline;

let crawler = CrawlerBuilder::new(MySpider)
    .add_pipeline(JsonPipeline::new("output.json")?)
    .add_pipeline(ConsolePipeline::new())
    .build()
    .await?;

Modules

console

Item Pipeline for writing scraped items to the console.

dedup

Item Pipeline for deduplicating scraped items.

pipeline

Trait for defining item processing pipelines in spider-pipeline.

transform

Item Pipeline for transforming scraped items.

validation

Item Pipeline for validating scraped items.