incpush
Incrementally back up a big local folder to object storage — uploading only changed files and preserving all previous versions — without ever listing the destination. Scales to millions of files and supports multiple storage backends (GCS, S3, Azure, local).
Why not gsutil rsync / aws s3 sync?
Those diff by listing the entire destination on every run, which costs a
Class A request per object — expensive when you have many files but few
changes. incpush never reads the destination to diff. It keeps a local
manifest of what it has already uploaded and compares against that, so a run
over 10M unchanged files makes zero destination requests.
Install
Produces a single self-contained incpush binary (no runtime dependencies).
Usage
# examples
Authentication
Credentials are read from the standard per-provider environment variables —
the destination URL scheme selects the backend. Set the relevant variables (in
your shell, .env, systemd unit, CI secret, etc.) before running.
Amazon S3 — s3://bucket/prefix/
# or AWS_REGION
# optional (temporary credentials)
On EC2/ECS/EKS you can omit the keys and let the instance/task IAM role or web
identity be used automatically (AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE,
or the container credentials endpoint).
S3-compatible (MinIO, Cloudflare R2, Backblaze B2, Wasabi, …) — s3://bucket/prefix/
Same as S3, plus point it at the custom endpoint:
# many providers accept "auto"
# only for plain-HTTP endpoints (e.g. local MinIO)
Google Cloud Storage — gs://bucket/prefix/
# a service-account key file:
# or GOOGLE_SERVICE_ACCOUNT_PATH
# or the key JSON inline:
# or Application Default Credentials:
GOOGLE_APPLICATION_CREDENTIALS is also what gcloud auth application-default login writes, and GCE/GKE metadata-server credentials are used automatically
when no variable is set.
Microsoft Azure Blob — az://container/prefix/
# one of the following credential types:
# shared key
# SAS token
# or a service principal:
Local / second disk — file:///mnt/disk2/backup/
No credentials needed.
Options
| Flag | Default | Description |
|---|---|---|
--diff <mtime|hash|full-hash> |
mtime |
Change-detection strategy (see below). |
-p, --print |
Dry run: show what would upload, write nothing. | |
-q, --quiet |
Don't print each uploaded file. | |
--reconcile |
Detect + record deletions this run (full manifest sweep). | |
--no-adopt |
Disable first-run adoption; re-upload everything even if the destination already has matching objects. | |
--concurrency <N> |
64 |
Concurrent uploads. |
--retries <N> |
3 |
Retry attempts per file. |
--snapshot-every <N> |
100 |
Upload a full manifest snapshot every N runs. |
--zstd-level <N> |
6 |
Compression level for deltas/snapshots. |
--exclude <glob> |
Exclude matching paths (repeatable). | |
--no-backupignore |
Don't honor per-directory .backupignore files (skips a per-directory probe that is costly on trees with millions of folders). |
Adopting a folder already backed up by another tool
If the destination already holds objects from a previous tool (rclone, gsutil rsync, …) and there's no incpush manifest yet, the first run adopts
them automatically instead of re-uploading. It lists the destination once and,
for each local file, reuses the remote object when it matches by path +
size and the local file wasn't modified after the remote copy's timestamp —
uploading only what's missing or changed. Adopted files are recorded in the
manifest with their local mtime, so subsequent runs diff normally.
This is free when there's nothing to adopt: an empty destination is a single list
call that returns nothing. Adoption matches on size + timestamp, not content
hashes, so it trusts that same-path / same-size / not-newer files are identical —
correct for the normal single-source migration. Pass --no-adopt to force a
full re-upload instead (e.g. if you suspect the existing remote data is wrong).
How change detection works
mtime(default): compare size + mtime only. No file contents are read — just onestatper file. This is the scale path for 1–10M files.hash: cheap size/mtime filter first, then crc32c only the candidates that already look changed — suppresses re-uploading files that were merelytouched. Hashing never touches the unchanged majority.full-hash: crc32c every file. Integrity audits only; reads every byte.
State layout
The manifest is log-structured (LSM): a compact sorted base plus tiny per-run segments, so a single changed file never rewrites the whole manifest.
Local (in the source folder, git-ignored):
.incpush/
manifest/base.fcm # compact, front-coded, sorted base
manifest/seg-NNNNN.fcm # per-run change segments (upserts + deletion tombstones)
state.json # last completed run number
Diffs merge base + segments newest-wins into one sorted stream and merge-join it against the (also sorted) walk — all streamed, so memory stays flat. Segments are compacted back into the base periodically. The walk itself is a sorted depth-first traversal, so it emits paths in the manifest's canonical order with no separate global sort.
Remote (under a reserved prefix in the bucket):
.incpush/
HEAD.json # tiny pointer to current state
snapshots/manifest-<N>.fcm # full compacted snapshot (every --snapshot-every runs)
deltas/run-<N>.delta.zst # per-run change log (KB-sized): only changed/deleted files
Each run ships a small delta (only the changed files), not the whole manifest. A fresh machine reconstructs by downloading the base snapshot and replaying deltas; this happens automatically when no local manifest is present.
Deletions are never destructive remotely: --reconcile only stops tracking a
removed path in the manifest — the bucket keeps every previously uploaded
version.
incpush also never modifies, moves, or deletes anything in your source folder
— it only reads files and writes its own .incpush/ state directory.
Note on non-ASCII filenames
Object keys go through object_store, which percent-encodes non-ASCII bytes in
the remote key (e.g. café.txt is stored as caf%C3%A9.txt). ASCII names,
including spaces, are stored verbatim. This is deterministic and reversible, and
incpush keys its own manifest on the real local path, so diffing and restore
through incpush stay faithful — but the raw object name in the bucket is encoded.
--adopt is unaffected: it percent-decodes remote keys before matching, so a
unicode-named file already in the bucket (whether stored raw or encoded) is still
recognized and not re-uploaded.
Development
License
MIT