Crate millipede_storage_memory 

millipede-storage-memory

In-memory StorageClient implementation for the Millipede web crawler.

This crate provides Millipede’s default storage backend: process-local datasets, key-value stores, and a lease-based request queue with deduplication.

Installation

[dependencies]
millipede-storage-memory = "0.1"

Most users should depend on the umbrella millipede crate instead.

Example

use millipede_core::prelude::{AddOptions, Request, RequestQueue};
use millipede_storage_memory::MemoryRequestQueue;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let queue = MemoryRequestQueue::new("example");
    let request = Request::get("https://example.com/").build()?;

    queue.add(request, AddOptions::default()).await?;
    let lease = queue.fetch_next().await?.expect("request was queued");
    queue.mark_handled(lease).await?;

    assert!(queue.is_finished().await?);
    Ok(())
}

Part of Millipede

See the Millipede guide for request queues, datasets, key-value stores, and backend selection.

License

Licensed under either MIT OR Apache-2.0 at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate is dual-licensed as above, without any additional terms or conditions.

Modules

prelude

Commonly used items from this crate.

Structs

DomainRoundRobin

A pure frontier that rotates fairly between URL hosts.

MemoryDataset

An in-process, append-only JSON dataset.

MemoryKeyValueStore

An in-process byte-oriented key-value store.

MemoryRequestQueue

A thread-safe, lease-based request queue stored entirely in memory.

MemoryStorageClient

An in-process storage client that shares named stores across open calls.

Enums

MemoryQueuePolicy

Ordering policy used by an in-memory request queue.