Struct HtmlKind 

pub struct HtmlKind { /* private fields */ }

HTML fetching behavior that delegates transport concerns to HttpKind.

Implementations

impl HtmlKind

pub fn builder() -> HtmlKindBuilder

Starts configuring an HTML crawler kind.

pub fn new() -> Result<Self, HttpClientError>

Creates an HTML kind with all defaults.

pub fn from_http(http: HttpKind) -> Self

Wraps an already configured HTTP kind.

Trait Implementations

impl CrawlerKind for HtmlKind

type Context = HtmlContext

The context passed to user handlers and lifecycle hooks.

fn start<'a>( &'a self, env: &'a CrawlerEnv, ) -> BoxFuture<'a, Result<(), CrawlError>>

Runs once before the crawler fetches any request.

fn before_request<'a>( &'a self, prep: &'a mut RequestPrep, ) -> BoxFuture<'a, Result<(), CrawlError>>

Mutates a request before an attempt executes.

fn execute<'a>( &'a self, env: RequestEnv<'a>, ) -> BoxFuture<'a, Result<Self::Context, CrawlError>>

Executes one request attempt and constructs its handler context.

fn observe(&self, ctx: &Self::Context) -> AttemptObservation

Called once after execute() succeeds; feeds statistics, HandledRequest, and RetryStrategy metadata.

fn after_success<'a>( &'a self, ctx: &'a mut Self::Context, ) -> BoxFuture<'a, Result<(), CrawlError>>

Runs after the user handler succeeds.

fn cleanup( &self, outcome: RequestOutcome<Self::Context>, ) -> BoxFuture<'_, Result<(), CrawlError>>

Runs after every attempt concludes, regardless of its outcome.

fn stop<'a>( &'a self, env: &'a CrawlerEnv, ) -> BoxFuture<'a, Result<(), CrawlError>>

Runs once when crawler shutdown begins.