incpush 0.1.2

Incrementally back up a big local folder to object storage (GCS/S3/Azure), uploading only changed files and preserving previous versions
Documentation

incpush

Incrementally back up a big local folder to object storage — uploading only changed files and preserving all previous versions — without ever listing the destination. Scales to millions of files and supports multiple storage backends (GCS, S3, Azure, local).

Why not gsutil rsync / aws s3 sync?

Those diff by listing the entire destination on every run, which costs a Class A request per object — expensive when you have many files but few changes. incpush never reads the destination to diff. It keeps a local manifest of what it has already uploaded and compares against that, so a run over 10M unchanged files makes zero destination requests.

Install

cargo install incpush

Produces a single self-contained incpush binary (no runtime dependencies).

Usage

incpush <source> <destination-url> [options]

# examples
incpush ./photos gs://my-bucket/photos/
incpush ./data  s3://my-bucket/backups/data/
incpush ./data  file:///mnt/disk2/backup/     # local/second-disk backend

Authentication

Credentials are read from the standard per-provider environment variables — the destination URL scheme selects the backend. Set the relevant variables (in your shell, .env, systemd unit, CI secret, etc.) before running.

Amazon S3 — s3://bucket/prefix/

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_DEFAULT_REGION=us-east-1        # or AWS_REGION
export AWS_SESSION_TOKEN=...               # optional (temporary credentials)

On EC2/ECS/EKS you can omit the keys and let the instance/task IAM role or web identity be used automatically (AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE, or the container credentials endpoint).

S3-compatible (MinIO, Cloudflare R2, Backblaze B2, Wasabi, …) — s3://bucket/prefix/

Same as S3, plus point it at the custom endpoint:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com   # your endpoint
export AWS_DEFAULT_REGION=auto            # many providers accept "auto"
export AWS_ALLOW_HTTP=true                # only for plain-HTTP endpoints (e.g. local MinIO)

Google Cloud Storage — gs://bucket/prefix/

# a service-account key file:
export GOOGLE_SERVICE_ACCOUNT=/path/to/key.json      # or GOOGLE_SERVICE_ACCOUNT_PATH
# or the key JSON inline:
export GOOGLE_SERVICE_ACCOUNT_KEY='{"type":"service_account", ...}'
# or Application Default Credentials:
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/adc.json

GOOGLE_APPLICATION_CREDENTIALS is also what gcloud auth application-default login writes, and GCE/GKE metadata-server credentials are used automatically when no variable is set.

Microsoft Azure Blob — az://container/prefix/

export AZURE_STORAGE_ACCOUNT_NAME=...
# one of the following credential types:
export AZURE_STORAGE_ACCOUNT_KEY=...                 # shared key
export AZURE_STORAGE_SAS_KEY=...                     # SAS token
# or a service principal:
export AZURE_STORAGE_CLIENT_ID=...
export AZURE_STORAGE_CLIENT_SECRET=...
export AZURE_STORAGE_TENANT_ID=...

Local / second disk — file:///mnt/disk2/backup/

No credentials needed.

Options

Flag Default Description
--diff <mtime|hash|full-hash> mtime Change-detection strategy (see below).
-p, --print Dry run: show what would upload, write nothing.
-q, --quiet Don't print each uploaded file.
--reconcile Detect + record deletions this run (full manifest sweep).
--no-adopt Disable first-run adoption; re-upload everything even if the destination already has matching objects.
--concurrency <N> 64 Concurrent uploads.
--retries <N> 3 Retry attempts per file.
--snapshot-every <N> 100 Upload a full manifest snapshot every N runs.
--zstd-level <N> 6 Compression level for deltas/snapshots.
--exclude <glob> Exclude matching paths (repeatable). Also honors .backupignore.

Adopting a folder already backed up by another tool

If the destination already holds objects from a previous tool (rclone, gsutil rsync, …) and there's no incpush manifest yet, the first run adopts them automatically instead of re-uploading. It lists the destination once and, for each local file, reuses the remote object when it matches by path + size and the local file wasn't modified after the remote copy's timestamp — uploading only what's missing or changed. Adopted files are recorded in the manifest with their local mtime, so subsequent runs diff normally.

This is free when there's nothing to adopt: an empty destination is a single list call that returns nothing. Adoption matches on size + timestamp, not content hashes, so it trusts that same-path / same-size / not-newer files are identical — correct for the normal single-source migration. Pass --no-adopt to force a full re-upload instead (e.g. if you suspect the existing remote data is wrong).

How change detection works

  • mtime (default): compare size + mtime only. No file contents are read — just one stat per file. This is the scale path for 1–10M files.
  • hash: cheap size/mtime filter first, then crc32c only the candidates that already look changed — suppresses re-uploading files that were merely touched. Hashing never touches the unchanged majority.
  • full-hash: crc32c every file. Integrity audits only; reads every byte.

State layout

The manifest is log-structured (LSM): a compact sorted base plus tiny per-run segments, so a single changed file never rewrites the whole manifest.

Local (in the source folder, git-ignored):

.incpush/
  manifest/base.fcm        # compact, front-coded, sorted base
  manifest/seg-NNNNN.fcm   # per-run change segments (upserts + deletion tombstones)
  state.json               # last completed run number

Diffs merge base + segments newest-wins into one sorted stream and merge-join it against the (also sorted) walk — all streamed, so memory stays flat. Segments are compacted back into the base periodically. The walk itself is a sorted depth-first traversal, so it emits paths in the manifest's canonical order with no separate global sort.

Remote (under a reserved prefix in the bucket):

.incpush/
  HEAD.json                       # tiny pointer to current state
  snapshots/manifest-<N>.fcm      # full compacted snapshot (every --snapshot-every runs)
  deltas/run-<N>.delta.zst        # per-run change log (KB-sized): only changed/deleted files

Each run ships a small delta (only the changed files), not the whole manifest. A fresh machine reconstructs by downloading the base snapshot and replaying deltas; this happens automatically when no local manifest is present.

Deletions are never destructive remotely: --reconcile only stops tracking a removed path in the manifest — the bucket keeps every previously uploaded version.

incpush also never modifies, moves, or deletes anything in your source folder — it only reads files and writes its own .incpush/ state directory.

Note on non-ASCII filenames

Object keys go through object_store, which percent-encodes non-ASCII bytes in the remote key (e.g. café.txt is stored as caf%C3%A9.txt). ASCII names, including spaces, are stored verbatim. This is deterministic and reversible, and incpush keys its own manifest on the real local path, so diffing and restore through incpush stay faithful — but the raw object name in the bucket is encoded.

--adopt is unaffected: it percent-decodes remote keys before matching, so a unicode-named file already in the bucket (whether stored raw or encoded) is still recognized and not re-uploaded.

Development

cargo test        # unit + end-to-end tests (the e2e suite runs against a file:// backend)
cargo build --release

License

MIT