Crate millipede_storage_fs 

millipede-storage-fs

File-system StorageClient implementation for the Millipede web crawler.

This crate persists Millipede datasets, key-value stores, and request queues beneath a configurable local directory.

Data-loss warning: purge_on_start defaults to true. Starting a crawler with its default configuration purges datasets, request queues, and non-INPUT key-value-store records in the selected storage directory.

The on-disk layout is compatible with Crawlee’s ./storage directory; read the Crawlee storage migration guide before opening existing data.

Installation

[dependencies]
millipede-storage-fs = "0.1"

Most users should depend on the umbrella millipede crate with its file-system storage feature instead.

Example

use millipede_core::prelude::{DatasetExt, StorageClient};
use millipede_storage_fs::FsStorageClient;
use serde_json::json;

let directory = tempfile::tempdir()?;
let storage = FsStorageClient::new(directory.path());
let dataset = storage.open_dataset(None).await?;
dataset.push(&json!({ "url": "https://example.com/" })).await?;

Part of Millipede

See the Millipede guide for storage configuration, persistence, and migration guidance.

License

Licensed under either MIT OR Apache-2.0 at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate is dual-licensed as above, without any additional terms or conditions.

Modules

prelude

Commonly used items from this crate.

Structs

FsDataset

A file-system-backed, append-only JSON dataset.

FsKeyValueStore

A file-system-backed byte-oriented key-value store.

FsRequestQueue

A durable FIFO request queue backed by atomic file replacements.

FsStorageClient

A file-system storage client using Crawlee-compatible directory layouts.