hibp-sync-client

Persistent daemon that keeps a local HIBP sha1t48 dataset in sync with an hibp-bin-fetch serve instance. Wakes nightly at a configurable UTC time, pulls whichever prefix files changed since the last sync (or all of them on first run), then sleeps until the following night.

This is the client counterpart to hibp-bin-fetch serve. Use it to keep a local replica of the HIBP dataset up to date, then check passwords against it with hibp-verifier.

Features

Runs as a persistent daemon with a configurable nightly sync time
Full sync on first run, delta sync on subsequent runs
Segmented sequential downloads with configurable resume granularity
Automatic fallback from delta to full sync when the server has advanced multiple cycles
Crash-safe: interrupted syncs resume where they left off; stale staging is discarded if the server has moved on
Sync failures are logged and skipped - the daemon continues to the next nightly cycle

Installation

cargo install hibp-sync-client

Usage

First, ensure an hibp-bin-fetch serve instance is running:

hibp-bin-fetch serve --data-dir /var/lib/hibp-sync --listen 0.0.0.0:8765

Then run the daemon against it (syncs nightly at 04:00 UTC by default):

hibp-sync-client --server-url http://192.168.1.10:8765 --data-dir ./hibp-data

Sync immediately on startup, then continue on the nightly schedule:

hibp-sync-client --server-url http://192.168.1.10:8765 --data-dir ./hibp-data --sync-on-start

Sync at a custom time and with finer resume granularity:

hibp-sync-client --server-url http://192.168.1.10:8765 --data-dir ./hibp-data \
    --sync-at 05:30 --segments 32

Options

Flag	Default	Description
`--server-url`	required	URL of the hibp-bin-fetch serve instance
`--data-dir`	required	Directory where .bin files are stored
`--sync-at`	`04:00`	UTC time to run the nightly sync cycle (HH:MM)
`--sync-on-start`	off	Run a sync cycle immediately before entering the schedule
`--segments`	`16`	Number of segments to split the sync into
`--log-level`	`info`	Log verbosity: error, warn, info, debug, trace

Sync Modes

Full Sync

On first run (no local dataset), the server divides the entire 1,048,576-prefix dataset into --segments zstd-compressed chunks. The client fetches them sequentially, unpacking each segment and writing the prefix files to a staging directory before atomically moving them to the data directory.

Delta Sync

On subsequent runs the client checks whether the server's prev_last_updated timestamp matches the local last_updated. If it does, only the prefixes that changed in the server's most recent nightly cycle are transferred - typically a few thousand files rather than the full 1 million. If the server has advanced more than one cycle the client falls back to a full sync automatically.

Crash-Safe Design

Downloads are staged in .staging/ within the data directory. Each segment is marked complete with a sentinel file once written, so a restart skips already-finished segments. After all segments are done a .complete marker is written; the next run will finish the atomic move even if the process died before the commit completed. If the server's data has changed since staging began, the stale staging directory is discarded and a fresh sync starts.

Library Usage

The sync function can be called directly when embedding sync logic in a larger application:

use hibp_sync_client::sync::{Config, Outcome, sync};
use std::path::PathBuf;
use url::Url;

let config = Config {
    server_url: Url::parse("http://192.168.1.10:8765").unwrap(),
    data_dir: PathBuf::from("./hibp-data"),
    segments: 16,
};

match sync(&config).await? {
    Outcome::UpToDate => println!("already up to date"),
    Outcome::DeltaSync { changed_count } => println!("{changed_count} files updated"),
    Outcome::FullSync { file_count } => println!("{file_count} files written"),
}

Related Projects

hibp-bin-fetch - Downloads the HIBP dataset and runs the serve instance this client connects to
hibp-verifier - Checks passwords against the synced dataset

License

MIT

hibp-sync-client 0.1.1