rightkit-sync 0.2.3

Brand-neutral desktop/mobile cloud sync engine: change tokens, tombstones, last-writer-wins conflicts, resumable batches, offline queue, retry/backoff.
Documentation
# rightkit-sync

Generic cloud sync engine for Right Suite apps. It owns change cursors, tombstones, last-writer-wins conflict resolution, resumable batches, an offline queue, retry and backoff, and an account-scoped ledger. It is aligned with the HeardRight iOS CloudKit record and conflict model.

**Use it** when an app must sync records through a cloud store and needs the same tombstone, conflict, and retry rules as the other Right Suite apps.

**Do not use it** as a transport on its own. The engine takes adapter traits; the CloudKit backend is the macOS helper described below.

## Public surface

- `API.md` (in this crate): adapter traits, the shared wire contract, the helper protocol, and per-app migration. Read it before changing record or conflict behavior.
- `helper::{CloudKitHelper, HelperBackend, HelperAuth}`: the macOS CloudKit backend, over the Swift `rightkit-cloudkit-helper` (`swift/RightKitSwift` at the RightKit root). `HelperAuth` is defined in `src/helper/mod.rs`.

## Features

| Feature | Default | Adds |
|---|---|---|
| `helper` | yes | the `helper` module: the CloudKit helper-process backend (`rightkit-sync-helper/1`). No dependency of its own; turn it off on hosts without a helper (Android). |
| `process-usage` | yes | `SyncStatus::process` and the `rightkit-child-process` dependency. Turn it off on Android. The pass counters in `SyncStatus::metrics` stay. |
| `cipher` | no | `cipher::{RecordCipher, EncryptingBackend, KeyProvider, StaticKeys, CipherKey}`: end-to-end encryption of record payloads (ChaCha20-Poly1305, HKDF-SHA256 from `ring`; `zeroize`). |
| `http` | no | **Supported** (ruling rightkit#967): the hosted sync service is live, `packages/sync-worker` is deployed for HeardRight and is its reference server. Apps may enable it for the hosted service, and a later self-hosted deployment of the same Worker uses the same protocol. `http::{HttpBackend, HttpAuth, TokenSource}`: blocking client of the `rightkit-sync-http/1` protocol (`HTTP-PROTOCOL.md`), over `rightkit-http`'s blocking client. |
| `vault` | no | The `vault` module (implies `cipher`): conditional-object vault sync for password-vault data, which the last-writer-wins engine and its plaintext `pending_apply` cannot carry. See "Conditional-object vault" below. |
| `vault-s3` | no | `vault::s3::S3ConditionalStore`: Cloudflare R2 and S3-compatible buckets, Signature V4 over `rightkit-http`; adds `vault` and `rightkit-http`. |
| `drive` | no | The non-Apple backend (CloudKit is the Apple one): `drive::{DriveBackend, DriveAuth, DriveTokenSource, FnToken, DriveConfig}`, own-cloud sync through the user's Google Drive `appDataFolder` (scope `drive.appdata`), over `rightkit-http`'s blocking client. See "Google Drive backend" below. |

Always: `serde`, `serde_json`, and `rightkit-fs` (`fs` feature, for the durable ledger write, `src/storage.rs`). The record wire encoding (`wire`) is shared by the helper and HTTP protocols.

A provider that encrypts (`HttpBackend` + `EncryptingBackend`) must run its engine with `SyncConfig::require_consent_on_scope_change` (`http::sync_config()` sets it) and an `account_scope` that includes the key id. Plaintext downloaded records the engine could not apply yet are kept in the ledger's `pending_apply`; the ledger file is owner-only but not encrypted.

Every deletion an encrypting client applies is authenticated: a tombstone the server serves without its sealed `enc` is quarantined (counted in `CipherStats::unauthenticated_tombstones`), never applied. A server may compact tombstones but must keep their `enc`. `HttpBackend::max_batch` defaults to 40 (the Cloudflare free-plan save cap); raise it only up to the server's `max_save_records`.

## Google Drive backend (feature `drive`)

`DriveBackend` syncs through the user's own Google Drive: one zone is a folder inside the hidden `appDataFolder`, one record is one small JSON file (the shared `wire` encoding), tombstones are kept as records, and the cursor is the Drive changes page token (polling only). No service of ours is involved and the data uses the user's own quota. It is pure Rust and shared by Windows, macOS (when not using CloudKit) and Android. The caller supplies the access token through `DriveTokenSource` (desktop from `rightkit-oauth`'s loopback PKCE flow, Android from the Kotlin `AuthorizationClient`); this crate does not depend on `rightkit-oauth`. Run the engine with `drive::sync_config()` (consent required on an account change) and a scope from `drive::account_scope`. The `cipher` wrapper stays optional on top.

**No atomic compare-and-swap.** Google's Drive v3 reference documents no precondition on `files.update` or `files.create` (no `If-Match`/ETag, no revision precondition, no create-only flag), and file names are not unique within a folder. `SavePolicy::IfServerUnchanged` is therefore a check (`version` of the file must equal the record's tag, else `SyncError::Conflict` with the server record) followed by the write, with a race window of about one request round trip (and the upload time of the record body). A write by another device that commits inside that window is overwritten by the later `PATCH`:

- a lost live edit heals on the next pass of the device that made it when its conflict policy says local wins (always the case under last-writer-wins when it was the newer edit);
- a tombstone overwritten inside the window leaves a live record on the server, so a device with a reset ledger can see a deleted item again (devices holding the tombstone ignore it);
- two devices creating one new record name at once are reduced by listing again after the create (smallest Drive file id wins, the other creator deletes its file and conflicts); if Drive's listing has not yet indexed the rival file both survive and the engine's duplicate reduction ranks them by `updatedAt`.

The exact analysis, the Drive endpoints used and the error mapping (401 to `Unavailable`, 403 rate limits, 429 and 5xx to `RetryAfter`, 404 to absent or `ZoneMissing`, other 4xx to `Unavailable`) are in the `drive` module docs. Removals seen in the change feed are never turned into deletions (a user clearing the app's hidden Drive data must not delete local data); a removed zone folder yields `ZoneMissing` and the engine republishes local items. What goes into a record (transcripts only? attachments?) and the per-record size cap (`DriveConfig::max_record_bytes`, 1 MiB) are decided by the app. Drive's real behaviour has not been exercised.

## Conditional-object vault (feature `vault`)

A password vault must never lose a secret edited on two devices. `vault::Vault` is a separate module, not a change to the record engine: it syncs one whole, sealed vault through an object store that documents atomic conditional writes (R2 and S3 document `If-Match` and `If-None-Match` on `PutObject` with strong consistency; Drive is excluded). The vault is a set of immutable sealed state objects, uploaded create-only (`If-None-Match: *`), and one sealed head that names the current one and is replaced only with `If-Match` (`If-None-Match: *` on the first write). A refused head write means another writer won: the loser re-reads, runs a three-way merge of base, local and remote, and writes again; after an uncertain write (timeout, transport error) the head is read back before deciding; retries are bounded (`VaultConfig::max_attempts`, default 8) and the call never sleeps. The consumer supplies the `MergePolicy` that decides what "the same secret edited twice" means (`KeepBoth` keeps the local value under the name and the remote value under `<name>.conflict-<8 hex>`; an edit against a deletion keeps the edit). The local ledger is sealed with the vault key, so it holds no plaintext, and everything in the bucket is sealed with `cipher::RecordCipher` (vault id as zone, object key as record name). A head older than one the device saw, a head that forks, or a head that vanished after it was seen is an error, never a fresh vault.

**Provider contract.** A `ConditionalStore` must make `put_if_absent` and `put_if_match` atomic against every other writer of the key, return a new ETag after every write, and give reads after a successful write the written value. `MemoryConditionalStore` is the in-process reference; `S3ConditionalStore` (feature `vault-s3`) is the R2/S3 adapter (no write retries inside the HTTP layer, a write that returns no ETag counts as unknown). Unreferenced state objects that lost a race stay in the bucket; expire them with a lifecycle rule. The two-writer race against a real bucket is `tests/vault_race_real_bucket.rs` (`#[ignore]`, needs `RIGHTKIT_VAULT_S3_*`); it has not been run.

## Scheduling and Android (WorkManager)

`sync_once(&mut engine, &SyncBudget, &SchedulePolicy) -> SyncReport` (module `schedule`, always compiled, platform-neutral) runs one bounded unit of sync and tells the host what to do next. It never sleeps and spawns no thread; it checks the elapsed budget between passes only (it cannot interrupt a running pass; the backend's HTTP timeouts bound that). The default budget is one pass. The engine's own backoff is kept in the ledger, so it survives the process being killed between WorkManager runs.

`SyncReport` carries `passes`, the last `outcome`, the summed `uploaded`/`downloaded`/`tombstones_*`/`quarantined`, `elapsed_ms`, a `work: WorkOutcome`, `next_delay_secs: Option<f64>` and `consent_required`. The pure helpers `next_delay`, `work_outcome` and `should_continue` are public and unit tested. Suggested delay: `idle_secs` (900) after `Synced`, the engine's figure after `Pending`, `Deferred` and failures (exponential 15 s doubling to 900 s, a server hint never shortened), never below `min_secs` (15), and `None` for `Disabled` and `ConsentRequired`.

**Kotlin call contract.** `rightkit-android` exposes only its own bridge (init, paths, Keystore, logcat); it exposes no other crate, so this crate is not added to it. The app's own cdylib (which links `rightkit-android`) wraps `sync_once` in its UniFFI surface, for example `fun syncOnce(maxPasses: UInt, maxElapsedMs: ULong): SyncReportFfi`, and the app's `CoroutineWorker`/`Worker` calls it:

1. `RightKit.init(context, ...)` ran in `Application.onCreate` (this also initialises the TLS verifier, see below). Build the engine once per process (a `DriveBackend` over a `rightkit-http` `Client`, `FileStorage` under `filesDir`).
2. In `doWork()` (a background thread, never the main thread), fetch a fresh Drive access token from `AuthorizationClient` and hand it to the `DriveTokenSource`; call `syncOnce`.
3. Map `work`: `Success` -> `Result.success()`, `Retry` -> `Result.retry()`, `Failure` -> `Result.failure()` (surface `consentRequired` in the UI and call `grant_consent`).
4. Enqueue the next run as a `OneTimeWorkRequest` with `setInitialDelay(nextDelaySecs)` (WorkManager's periodic minimum is 15 minutes and its `retry()` backoff is its own; a one-time chain keeps the engine's delay) and a network constraint. Do not enqueue when `nextDelaySecs` is `None`.

**TLS roots on Android (rightkit#149).** `rightkit-http` defaults to the OS verifier (`rustls-platform-verifier`), which on Android needs the `JavaVM` and `Context` initialised once per process before the first request (`rightkit-android`'s `platform-verifier` feature does it inside `RightKit.init`; the verifier's Kotlin artifact must be on the classpath). Until then requests fail closed. An app that has not wired it builds its `Client` with `TrustRoots::Bundled`. Build the crate for Android with `--no-default-features --features drive` (plus `cipher` if used). Nothing here is compiled for an Android target yet.

## Platform support

- macOS: the CloudKit helper backend. The helper is a separate Swift binary.
- Android: the `drive` backend (no cfgs in this crate; see "Scheduling and Android"). Not built for an Android target yet.
- Windows: `cfg(windows)` process handling in `src/helper/process.rs`.
- The engine is portable Rust.

Published version: see `INDEX.md` at the repository root. Changes are recorded in `CHANGELOG.md`.

## Example

Illustrative; not compiled as a doctest. For compiled usage, see, in the repository only, `tests/helper_e2e.rs` (the helper protocol against the real CloudKit container, `#[ignore]`d).

```rust,ignore
// Build a sync engine over CloudKitHelper in the app's macOS host, and drive it with the app's record mapper.
```

## Runtime budget

Generated from `runtime-budget.json`; edit that file, then run `node packages/release/runtime-budgets.mjs sync <crate-dir>`.

- Resident memory ceiling: 1024.0 MiB.
- CPU ceiling (idle, steady state): 1%.
- Wake-ups ceiling: 1 per second.
- Disk operations ceiling: 1 operations per second.
- Required scale: 100,000 records or more.
- Required hosts: macOS only (Adrian 2026-10-14 local-build grant (.rightkit-local-build.json); Windows covered by the next CI run).
- Status exposure: `SyncEngine::status()` in `src/engine.rs`; exposes resident memory, idle CPU, wake-ups, disk operations.
- Measured: darwin/arm64 (Mac18,5), rightkit-sync 0.2.2 at 5ab2baa539bc, 100,000 records (zone of 100000 Note records over MemoryBackend: full pull, 100 passes of 100 remote + 100 local edits, then 30 s idle), synthetic, `rustright cargo test -p rightkit-sync --profile dev --bench budget -- 100000 100 100 30`, 2026-10-18, evidence fa724e41ea92: resident memory 645.2 MiB, idle CPU 0%, wake-ups 0.033 per second, disk operations n/a (not measurable on this host).