Struct MockSite 

pub struct MockSite { /* private fields */ }

Available on crate feature testing only.

A Backend serving fixed MockPages by URI, for tests.

URIs without a page get a 404 Not Found. Statuses are treated as the real backends treat them, by a StatusPolicy: by default, an error status fails the fetch, so the crawler retries or fails the request.

Clones share the record of requests. Crawls with identities can be tested too, see sent.

Examples

use spire_core::crawler::Crawler;
use spire_core::frontier::seed::Seed;
use spire_core::testing::{MockPage, MockSite};
use spire_core::worker::from_fn;

let site = MockSite::new()
    .page("https://example.com/", MockPage::html("<p>Home</p>"))
    .page("https://example.com/old", MockPage::redirect("https://example.com/"));

let worker = from_fn(|cx: spire_core::Context<_>| async move { cx.body().await.map(drop) });
let crawler = Crawler::new(site.clone(), worker).with_seeds([
    Seed::get("https://example.com/old")?,
    Seed::get("https://example.com/missing")?,
]);

let stats = crawler.run().await?;
assert_eq!((stats.succeeded, stats.failed), (1, 1));
assert_eq!(site.requests().len(), 2);

Implementations

impl MockSite

pub fn new() -> Self

Creates a site without pages.

pub fn page(self, uri: &str, page: MockPage) -> Self

Serves page at uri, replacing any previous page there.

Panics

Panics if uri is not a valid absolute URI.

pub const fn with_status_policy(self, policy: StatusPolicy) -> Self

Replaces the StatusPolicy, Classify by default.

pub fn unreachable_through(self, name: impl Into<String>) -> Self

Fails every fetch through the identity named name, as a proxy refusing connections would: a retryable error without a status.

pub fn requests(&self) -> Vec<Uri>

Returns the URIs fetched so far, in order, including clones’ fetches. Redirects are followed within a fetch, so only the requested URI is listed.

pub fn sent(&self) -> Vec<Sent>

Returns the requests received so far, in order, with their identity and headers. Fetches through an unreachable identity are included.

pub fn visit(&self, seed: Seed) -> Visit

Prepares a Visit to seed, to run one request through a worker.

Trait Implementations

impl Backend for MockSite

fn connect_as( &self, identity: &Identity, ) -> impl Future<Output = Result<Self::Session>> + Send

Records each request’s identity, see MockSite::sent, and refuses those unreachable. Proxies are accepted, as nothing is sent through them.

type Session = MockSession

The session used to fetch a single request.

fn connect(&self) -> impl Future<Output = Result<Self::Session>> + Send

Returns a session for the next request, sent directly, from the crawler itself, with the backend’s own cookies and sessions.

impl Clone for MockSite

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MockSite

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for MockSite

fn default() -> Self

Returns the “default value” for a type.