rightkit-sync 0.4.1

Brand-neutral desktop/mobile cloud sync engine: change tokens, tombstones, last-writer-wins conflicts, resumable batches, offline queue, retry/backoff.
Documentation

rightkit-sync

Generic cloud sync engine for Right Suite apps. It owns change cursors, tombstones, last-writer-wins conflict resolution, resumable batches, an offline queue, retry and backoff, and an account-scoped ledger. It is aligned with the HeardRight iOS CloudKit record and conflict model.

Use it when an app must sync records through a cloud store and needs the same tombstone, conflict, and retry rules as the other Right Suite apps.

Do not use it as a transport on its own. The engine takes adapter traits; the CloudKit backend is the macOS helper described below.

Public surface

  • API.md (in this crate): adapter traits, the shared wire contract, the helper protocol, and per-app migration. Read it before changing record or conflict behavior.
  • helper::{CloudKitHelper, HelperBackend, HelperAuth}: the macOS CloudKit backend, over the Swift rightkit-cloudkit-helper (swift/RightKitSwift at the RightKit root). HelperAuth is defined in src/helper/mod.rs.

Features

Feature Default Adds
helper yes the helper module: the CloudKit helper-process backend (rightkit-sync-helper/1). No dependency of its own; turn it off on hosts without a helper (Android).
process-usage yes SyncStatus::process and the rightkit-child-process dependency. Turn it off on Android. The pass counters in SyncStatus::metrics stay.
cipher no cipher::{RecordCipher, EncryptingBackend, KeyProvider, StaticKeys, CipherKey}: end-to-end encryption of record payloads (ChaCha20-Poly1305, HKDF-SHA256 from ring; zeroize).
http no Supported (ruling rightkit#967): the hosted sync service is live, packages/sync-worker is deployed for HeardRight and is its reference server. Apps may enable it for the hosted service, and a later self-hosted deployment of the same Worker uses the same protocol. http::{HttpBackend, HttpAuth, TokenSource}: blocking client of the rightkit-sync-http/1 protocol (HTTP-PROTOCOL.md), over rightkit-http's blocking client.
vault no The vault module (implies cipher): conditional-object vault sync for password-vault data, which the last-writer-wins engine and its plaintext pending_apply cannot carry. See "Conditional-object vault" below.
vault-s3 no vault::s3::S3ConditionalStore: Cloudflare R2 and S3-compatible buckets, Signature V4 over rightkit-http; adds vault and rightkit-http.
drive no The non-Apple backend (CloudKit is the Apple one): drive::{DriveBackend, DriveAuth, DriveTokenSource, FnToken, DriveConfig}, own-cloud sync through the user's Google Drive appDataFolder (scope drive.appdata), over rightkit-http's blocking client. See "Google Drive backend" below.
transcripts no transcripts::{TranscriptSyncProfile, StateFileNames, CloudTranscript, TranscriptMapper, LocalVersions}: transcript records, stable ids, durable deletion outbox, consent scope gate, legacy ledger migration, CloudKit environment pin & retry classifiers. Adds rightkit-fs/digest. Existing hosts must supply their original identity domains & file names through TranscriptSyncProfile.

Always: serde, serde_json, and rightkit-fs (fs feature, for the durable ledger write, src/storage.rs). The record wire encoding (wire) is shared by the helper and HTTP protocols.

Transcript legacy migration imports ack seeds only with matching nonempty stored & consent-authorized scopes. Unscoped tombstones, pending deletions & aliases need matching ack ownership; scoped filenames must match current account's profile hash. Unproven sources stay intact & are excluded, allowing normal sync. Existing version entries retain their values. With proven seeds, unreadable or malformed versions return an error before destination writes, preserving migration retry after repair; missing versions start empty.

A provider that encrypts (HttpBackend + EncryptingBackend) must run its engine with SyncConfig::require_consent_on_scope_change (http::sync_config() sets it) and an account_scope that includes the key id. Plaintext downloaded records the engine could not apply yet are kept in the ledger's pending_apply; the ledger file is owner-only but not encrypted.

Every deletion an encrypting client applies is authenticated: a tombstone the server serves without its sealed enc is quarantined (counted in CipherStats::unauthenticated_tombstones), never applied. A server may compact tombstones but must keep their enc. HttpBackend::max_batch defaults to 40 (the Cloudflare free-plan save cap); raise it only up to the server's max_save_records.

Google Drive backend (feature drive)

DriveBackend syncs through the user's own Google Drive: one zone is a folder inside the hidden appDataFolder, one record is one small JSON file (the shared wire encoding), tombstones are kept as records, and the cursor is the Drive changes page token (polling only). No service of ours is involved and the data uses the user's own quota. It is pure Rust and shared by Windows, macOS (when not using CloudKit) and Android. The caller supplies the access token through DriveTokenSource (desktop from rightkit-oauth's loopback PKCE flow, Android from the Kotlin AuthorizationClient); this crate does not depend on rightkit-oauth. Run the engine with drive::sync_config() (consent required on an account change) and a scope from drive::account_scope. The cipher wrapper stays optional on top.

No atomic compare-and-swap. Google's Drive v3 reference documents no precondition on files.update or files.create (no If-Match/ETag, no revision precondition, no create-only flag), and file names are not unique within a folder. SavePolicy::IfServerUnchanged is therefore a check (version of the file must equal the record's tag, else SyncError::Conflict with the server record) followed by the write, with a race window of about one request round trip (and the upload time of the record body). A write by another device that commits inside that window is overwritten by the later PATCH:

  • a lost live edit heals on the next pass of the device that made it when its conflict policy says local wins (always the case under last-writer-wins when it was the newer edit);
  • a tombstone overwritten inside the window leaves a live record on the server, so a device with a reset ledger can see a deleted item again (devices holding the tombstone ignore it);
  • two devices creating one new record name at once are reduced by listing again after the create (smallest Drive file id wins, the other creator deletes its file and conflicts); if Drive's listing has not yet indexed the rival file both survive and the engine's duplicate reduction ranks them by updatedAt.

The exact analysis, the Drive endpoints used and the error mapping (401 to Unavailable, 403 rate limits, 429 and 5xx to RetryAfter, 404 to absent or ZoneMissing, other 4xx to Unavailable) are in the drive module docs. Removals seen in the change feed are never turned into deletions (a user clearing the app's hidden Drive data must not delete local data); a removed zone folder yields ZoneMissing and the engine republishes local items. What goes into a record (transcripts only? attachments?) and the per-record size cap (DriveConfig::max_record_bytes, 1 MiB) are decided by the app. Drive's real behaviour has not been exercised.

Conditional-object vault (feature vault)

A password vault must never lose a secret edited on two devices. vault::Vault is a separate module, not a change to the record engine: it syncs one whole, sealed vault through an object store that documents atomic conditional writes (R2 and S3 document If-Match and If-None-Match on PutObject with strong consistency; Drive is excluded). The vault is a set of immutable sealed state objects, uploaded create-only (If-None-Match: *), and one sealed head that names the current one and is replaced only with If-Match (If-None-Match: * on the first write). A refused head write means another writer won: the loser re-reads, runs a three-way merge of base, local and remote, and writes again; after an uncertain write (timeout, transport error) the head is read back before deciding; retries are bounded (VaultConfig::max_attempts, default 8) and the call never sleeps. The consumer supplies the MergePolicy that decides what "the same secret edited twice" means (KeepBoth keeps the local value under the name and the remote value under <name>.conflict-<8 hex>; an edit against a deletion keeps the edit). The local ledger is sealed with the vault key, so it holds no plaintext, and everything in the bucket is sealed with cipher::RecordCipher (vault id as zone, object key as record name). A head older than one the device saw, a head that forks, or a head that vanished after it was seen is an error, never a fresh vault.

Provider contract. A ConditionalStore must make put_if_absent and put_if_match atomic against every other writer of the key, return a new ETag after every write, and give reads after a successful write the written value. MemoryConditionalStore is the in-process reference; S3ConditionalStore (feature vault-s3) is the R2/S3 adapter (no write retries inside the HTTP layer, a write that returns no ETag counts as unknown). Unreferenced state objects that lost a race stay in the bucket; expire them with a lifecycle rule. The two-writer race against a real bucket is tests/vault_race_real_bucket.rs (#[ignore], needs RIGHTKIT_VAULT_S3_*); it has not been run.

Scheduling and Android (WorkManager)

sync_once(&mut engine, &SyncBudget, &SchedulePolicy) -> SyncReport (module schedule, always compiled, platform-neutral) runs one bounded unit of sync and tells the host what to do next. It never sleeps and spawns no thread; it checks the elapsed budget between passes only (it cannot interrupt a running pass; the backend's HTTP timeouts bound that). The default budget is one pass. The engine's own backoff is kept in the ledger, so it survives the process being killed between WorkManager runs.

SyncReport carries passes, the last outcome, the summed uploaded/downloaded/tombstones_*/quarantined, elapsed_ms, a work: WorkOutcome, next_delay_secs: Option<f64> and consent_required. The pure helpers next_delay, work_outcome and should_continue are public and unit tested. Suggested delay: idle_secs (900) after Synced, the engine's figure after Pending, Deferred and failures (exponential 15 s doubling to 900 s, a server hint never shortened), never below min_secs (15), and None for Disabled and ConsentRequired.

Kotlin call contract. rightkit-android exposes only its own bridge (init, paths, Keystore, logcat); it exposes no other crate, so this crate is not added to it. The app's own cdylib (which links rightkit-android) wraps sync_once in its UniFFI surface, for example fun syncOnce(maxPasses: UInt, maxElapsedMs: ULong): SyncReportFfi, and the app's CoroutineWorker/Worker calls it:

  1. RightKit.init(context, ...) ran in Application.onCreate (this also initialises the TLS verifier, see below). Build the engine once per process (a DriveBackend over a rightkit-http Client, FileStorage under filesDir).
  2. In doWork() (a background thread, never the main thread), fetch a fresh Drive access token from AuthorizationClient and hand it to the DriveTokenSource; call syncOnce.
  3. Map work: Success -> Result.success(), Retry -> Result.retry(), Failure -> Result.failure() (surface consentRequired in the UI and call grant_consent).
  4. Enqueue the next run as a OneTimeWorkRequest with setInitialDelay(nextDelaySecs) (WorkManager's periodic minimum is 15 minutes and its retry() backoff is its own; a one-time chain keeps the engine's delay) and a network constraint. Do not enqueue when nextDelaySecs is None.

TLS roots on Android (rightkit#149). rightkit-http defaults to the OS verifier (rustls-platform-verifier), which on Android needs the JavaVM and Context initialised once per process before the first request (rightkit-android's platform-verifier feature does it inside RightKit.init; the verifier's Kotlin artifact must be on the classpath). Until then requests fail closed. An app that has not wired it builds its Client with TrustRoots::Bundled. Build the crate for Android with --no-default-features --features drive (plus cipher if used). Nothing here is compiled for an Android target yet.

Platform support

  • macOS: the CloudKit helper backend. The helper is a separate Swift binary.
  • Android: the drive backend (no cfgs in this crate; see "Scheduling and Android"). Not built for an Android target yet.
  • Windows: cfg(windows) process handling in src/helper/process.rs.
  • The engine is portable Rust.

Published version: see INDEX.md at the repository root. Changes are recorded in CHANGELOG.md.

Example

Illustrative; not compiled as a doctest. For compiled usage, see, in the repository only, tests/helper_e2e.rs (the helper protocol against the real CloudKit container, #[ignore]d).

// Build a sync engine over CloudKitHelper in the app's macOS host, and drive it with the app's record mapper.

Runtime budget

Generated from runtime-budget.json; edit that file, then run node packages/release/runtime-budgets.mjs sync <crate-dir>.

  • Resident memory ceiling: 1024.0 MiB.
  • CPU ceiling (idle, steady state): 1%.
  • Wake-ups ceiling: 1 per second.
  • Disk operations ceiling: 1 operations per second.
  • Required scale: 100,000 records or more.
  • Required hosts: macOS only (Adrian 2026-10-14 local-build grant (.rightkit-local-build.json); Windows covered by the next CI run).
  • Status exposure: SyncEngine::status() in src/engine.rs; exposes resident memory, idle CPU, wake-ups, disk operations.
  • Measured: darwin/arm64 (Mac18,5), rightkit-sync 0.2.2 at 5ab2baa539bc, 100,000 records (zone of 100000 Note records over MemoryBackend: full pull, 100 passes of 100 remote + 100 local edits, then 30 s idle), synthetic, rustright cargo test -p rightkit-sync --profile dev --bench budget -- 100000 100 100 30, 2026-10-18, evidence fa724e41ea92: resident memory 645.2 MiB, idle CPU 0%, wake-ups 0.033 per second, disk operations n/a (not measurable on this host).

Desktop transcript host (transcript-host)

transcript-host enables transcripts, helper, process-usage & Tokio (sync, time, rt). transcript_host::TranscriptSync owns consent transitions, account authorization, engine lifetime, guarded ledger writes, scoped deletion outbox & event-driven scheduling. macOS hosts use transcript_host::cloudkit::CloudKitTransport; other transports implement SyncTransport. Apps provide TranscriptHost, windowed HistorySession, runtime spawner, state paths, identity values, copy & scheduler timings. Start with TranscriptSyncConfig, CloudKitConfig & trait contracts in rustdoc.

Host runtime needs its time driver enabled. Construct TranscriptSync, call start once runtime runs, notify durable edits through notify_local_change, queue explicit deletions through enqueue_deleted_ids, then use set_enabled, status, deletion_backlog & delete_everywhere. History reads use windows; partial passes retain work, coalesce wakes & obey bounded retries. Ready requires live authorized account, matching durable ledger & fully completed current-epoch pass. Partial, paused, stalled-feed, account-change & unconfirmed states remain distinct.

Disabling first closes admission & stops transport, then verifies existing helper under transition serialization & independently attempts durable revoked fence. Failed stop proof reports NotConfirmed("stop_unverified"); failed fence persistence reports RevocationIncomplete; both report NotConfirmed("stop_unverified_revocation_incomplete"). Poisoned outbox mutex leaves outbox untouched while fence is still attempted. Exhausted epoch still permits revoked fence at its maximum value, skips outbox/consent rewrites & refuses future enables. Revoked fences reject every enabled consent regardless of epoch. Storage failure keeps scheduler stopped; retry set_enabled(false) to persist revocation.

Each deletion carries positively observed live account & durable consent epoch. Unverified, stale-epoch & other-account entries stay retained & inert. Engine hand-over records distinguish newly created tombstones from existing deletions; withdrawal preserves acknowledged tombstones. Legacy migration retains 0.3.1 account provenance rules & existing version entries. Capped no-follow reads reject oversized or linked state without replacing it. Version-1 outbox helpers refuse scoped version-2 files.

Built-in ledger writers (FileStorage, transcript host guard & migrate_legacy_once) use same adjacent <ledger>.lock with ten-second wait. Host compares durable generation & account under that lock before replacement; migration checks ledger absence under it. Custom LedgerStorage implementations & external writers must acquire same lock across validation & replacement. Generic FileStorage remains an unconditional byte store after lock acquisition; it supplies no account authorization or stale-view comparison. Use transcript host for those checks.

rightkit-child-process >= 0.5.12 supplies bundled-login-item resolution & expected-team signature verification. Existing rightkit-fs >= 0.4.15 floor covers no-follow reads, digest, locks & atomic replacement APIs; lock is enabled for built-in ledger writers. Qualification should cover consent/activation rollback, simultaneous supported writers, account changes, revoked-fence storage failures, exhausted epochs, outbox quarantine & histories spanning multiple passes against installed host.