hibp-sync-client 0.1.1

Syncs a local HIBP binary dataset from an hibp-bin-fetch serve instance
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127

# hibp-sync-client

[![CI](https://img.shields.io/github/actions/workflow/status/PrismaPhonic/hibp-rs/ci.yml?branch=main)](https://github.com/PrismaPhonic/hibp-rs/actions)
[![crates.io](https://img.shields.io/crates/v/hibp-sync-client.svg)](https://crates.io/crates/hibp-sync-client)
[![docs.rs](https://docs.rs/hibp-sync-client/badge.svg)](https://docs.rs/hibp-sync-client)
[![maintenance](https://img.shields.io/badge/maintenance-actively--developed-brightgreen)](https://github.com/PrismaPhonic/hibp-rs)

Persistent daemon that keeps a local HIBP sha1t48 dataset in sync with an
[hibp-bin-fetch](https://crates.io/crates/hibp-bin-fetch) serve instance. Wakes nightly at a
configurable UTC time, pulls whichever prefix files changed since the last sync (or all of
them on first run), then sleeps until the following night.

This is the client counterpart to `hibp-bin-fetch serve`. Use it to keep a local replica of
the HIBP dataset up to date, then check passwords against it with
[hibp-verifier](https://crates.io/crates/hibp-verifier).

## Features

- Runs as a persistent daemon with a configurable nightly sync time
- Full sync on first run, delta sync on subsequent runs
- Segmented sequential downloads with configurable resume granularity
- Automatic fallback from delta to full sync when the server has advanced multiple cycles
- Crash-safe: interrupted syncs resume where they left off; stale staging is discarded if the
  server has moved on
- Sync failures are logged and skipped - the daemon continues to the next nightly cycle

## Installation

```sh
cargo install hibp-sync-client
```

## Usage

First, ensure an `hibp-bin-fetch serve` instance is running:

```sh
hibp-bin-fetch serve --data-dir /var/lib/hibp-sync --listen 0.0.0.0:8765
```

Then run the daemon against it (syncs nightly at 04:00 UTC by default):

```sh
hibp-sync-client --server-url http://192.168.1.10:8765 --data-dir ./hibp-data
```

Sync immediately on startup, then continue on the nightly schedule:

```sh
hibp-sync-client --server-url http://192.168.1.10:8765 --data-dir ./hibp-data --sync-on-start
```

Sync at a custom time and with finer resume granularity:

```sh
hibp-sync-client --server-url http://192.168.1.10:8765 --data-dir ./hibp-data \
    --sync-at 05:30 --segments 32
```

### Options

| Flag              | Default  | Description                                          |
|-------------------|----------|------------------------------------------------------|
| `--server-url`    | required | URL of the hibp-bin-fetch serve instance             |
| `--data-dir`      | required | Directory where .bin files are stored                |
| `--sync-at`       | `04:00`  | UTC time to run the nightly sync cycle (HH:MM)       |
| `--sync-on-start` | off      | Run a sync cycle immediately before entering the schedule |
| `--segments`      | `16`     | Number of segments to split the sync into            |
| `--log-level`     | `info`   | Log verbosity: error, warn, info, debug, trace       |

## Sync Modes

### Full Sync

On first run (no local dataset), the server divides the entire 1,048,576-prefix dataset
into `--segments` zstd-compressed chunks. The client fetches them sequentially, unpacking
each segment and writing the prefix files to a staging directory before atomically moving
them to the data directory.

### Delta Sync

On subsequent runs the client checks whether the server's `prev_last_updated` timestamp
matches the local `last_updated`. If it does, only the prefixes that changed in the
server's most recent nightly cycle are transferred - typically a few thousand files rather
than the full 1 million. If the server has advanced more than one cycle the client falls
back to a full sync automatically.

## Crash-Safe Design

Downloads are staged in `.staging/` within the data directory. Each segment is marked
complete with a sentinel file once written, so a restart skips already-finished segments.
After all segments are done a `.complete` marker is written; the next run will finish the
atomic move even if the process died before the commit completed. If the server's data
has changed since staging began, the stale staging directory is discarded and a fresh
sync starts.

## Library Usage

The `sync` function can be called directly when embedding sync logic in a larger
application:

```rust,ignore
use hibp_sync_client::sync::{Config, Outcome, sync};
use std::path::PathBuf;
use url::Url;

let config = Config {
    server_url: Url::parse("http://192.168.1.10:8765").unwrap(),
    data_dir: PathBuf::from("./hibp-data"),
    segments: 16,
};

match sync(&config).await? {
    Outcome::UpToDate => println!("already up to date"),
    Outcome::DeltaSync { changed_count } => println!("{changed_count} files updated"),
    Outcome::FullSync { file_count } => println!("{file_count} files written"),
}
```

## Related Projects

- [hibp-bin-fetch](https://crates.io/crates/hibp-bin-fetch) - Downloads the HIBP dataset and runs the serve instance this client connects to
- [hibp-verifier](https://crates.io/crates/hibp-verifier) - Checks passwords against the synced dataset

## License

MIT