# incpush
Incrementally back up a big local folder to object storage — uploading **only
changed files** and **preserving all previous versions** — without ever listing
the destination. Scales to millions of files and supports multiple storage
backends (GCS, S3, Azure, local).
## Why not `gsutil rsync` / `aws s3 sync`?
Those diff by **listing the entire destination** on every run, which costs a
Class A request per object — expensive when you have many files but few
changes. `incpush` never reads the destination to diff. It keeps a **local
manifest** of what it has already uploaded and compares against that, so a run
over 10M unchanged files makes **zero** destination requests.
## Install
```sh
cargo install incpush
```
Produces a single self-contained `incpush` binary (no runtime dependencies).
## Usage
```sh
incpush <source> <destination-url> [options]
# examples
incpush ./photos gs://my-bucket/photos/
incpush ./data s3://my-bucket/backups/data/
incpush ./data file:///mnt/disk2/backup/ # local/second-disk backend
```
## Authentication
Credentials are read from the standard per-provider **environment variables** —
the destination URL scheme selects the backend. Set the relevant variables (in
your shell, `.env`, systemd unit, CI secret, etc.) before running.
### Amazon S3 — `s3://bucket/prefix/`
```sh
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_DEFAULT_REGION=us-east-1 # or AWS_REGION
export AWS_SESSION_TOKEN=... # optional (temporary credentials)
```
On EC2/ECS/EKS you can omit the keys and let the instance/task IAM role or web
identity be used automatically (`AWS_ROLE_ARN` + `AWS_WEB_IDENTITY_TOKEN_FILE`,
or the container credentials endpoint).
### S3-compatible (MinIO, Cloudflare R2, Backblaze B2, Wasabi, …) — `s3://bucket/prefix/`
Same as S3, plus point it at the custom endpoint:
```sh
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com # your endpoint
export AWS_DEFAULT_REGION=auto # many providers accept "auto"
export AWS_ALLOW_HTTP=true # only for plain-HTTP endpoints (e.g. local MinIO)
```
### Google Cloud Storage — `gs://bucket/prefix/`
```sh
# a service-account key file:
export GOOGLE_SERVICE_ACCOUNT=/path/to/key.json # or GOOGLE_SERVICE_ACCOUNT_PATH
# or the key JSON inline:
export GOOGLE_SERVICE_ACCOUNT_KEY='{"type":"service_account", ...}'
# or Application Default Credentials:
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/adc.json
```
`GOOGLE_APPLICATION_CREDENTIALS` is also what `gcloud auth application-default
login` writes, and GCE/GKE metadata-server credentials are used automatically
when no variable is set.
### Microsoft Azure Blob — `az://container/prefix/`
```sh
export AZURE_STORAGE_ACCOUNT_NAME=...
# one of the following credential types:
export AZURE_STORAGE_ACCOUNT_KEY=... # shared key
export AZURE_STORAGE_SAS_KEY=... # SAS token
# or a service principal:
export AZURE_STORAGE_CLIENT_ID=...
export AZURE_STORAGE_CLIENT_SECRET=...
export AZURE_STORAGE_TENANT_ID=...
```
### Local / second disk — `file:///mnt/disk2/backup/`
No credentials needed.
### Options
| `--diff <mtime\|hash\|full-hash>` | `mtime` | Change-detection strategy (see below). |
| `-p, --print` | | Dry run: show what would upload, write nothing. |
| `-q, --quiet` | | Don't print each uploaded file. |
| `--reconcile` | | Detect + record deletions this run (full manifest sweep). |
| `--no-adopt` | | Disable first-run adoption; re-upload everything even if the destination already has matching objects. |
| `--concurrency <N>` | `64` | Concurrent uploads. |
| `--retries <N>` | `3` | Retry attempts per file. |
| `--snapshot-every <N>` | `100` | Upload a full manifest snapshot every N runs. |
| `--zstd-level <N>` | `6` | Compression level for deltas/snapshots. |
| `--exclude <glob>` | | Exclude matching paths (repeatable). |
| `--no-backupignore` | | Don't honor per-directory `.backupignore` files (skips a per-directory probe that is costly on trees with millions of folders). |
## Adopting a folder already backed up by another tool
If the destination already holds objects from a previous tool (rclone, `gsutil
rsync`, …) and there's no `incpush` manifest yet, the **first run adopts
them automatically** instead of re-uploading. It lists the destination once and,
for each local file, **reuses** the remote object when it matches by **path +
size** and the local file **wasn't modified after** the remote copy's timestamp —
uploading only what's missing or changed. Adopted files are recorded in the
manifest with their local mtime, so subsequent runs diff normally.
This is free when there's nothing to adopt: an empty destination is a single list
call that returns nothing. Adoption matches on size + timestamp, not content
hashes, so it trusts that same-path / same-size / not-newer files are identical —
correct for the normal single-source migration. Pass **`--no-adopt`** to force a
full re-upload instead (e.g. if you suspect the existing remote data is wrong).
## How change detection works
- **`mtime` (default):** compare size + mtime only. No file contents are read —
just one `stat` per file. This is the scale path for 1–10M files.
- **`hash`:** cheap size/mtime filter first, then crc32c **only the candidates**
that already look changed — suppresses re-uploading files that were merely
`touch`ed. Hashing never touches the unchanged majority.
- **`full-hash`:** crc32c every file. Integrity audits only; reads every byte.
## State layout
The manifest is **log-structured** (LSM): a compact sorted base plus tiny
per-run segments, so a single changed file never rewrites the whole manifest.
Local (in the source folder, git-ignored):
```
.incpush/
manifest/base.fcm # compact, front-coded, sorted base
manifest/seg-NNNNN.fcm # per-run change segments (upserts + deletion tombstones)
state.json # last completed run number
```
Diffs merge base + segments newest-wins into one sorted stream and merge-join it
against the (also sorted) walk — all streamed, so memory stays flat. Segments are
compacted back into the base periodically. The walk itself is a **sorted
depth-first traversal**, so it emits paths in the manifest's canonical order with
no separate global sort.
Remote (under a reserved prefix in the bucket):
```
.incpush/
HEAD.json # tiny pointer to current state
snapshots/manifest-<N>.fcm # full compacted snapshot (every --snapshot-every runs)
deltas/run-<N>.delta.zst # per-run change log (KB-sized): only changed/deleted files
```
Each run ships a **small delta** (only the changed files), not the whole
manifest. A fresh machine reconstructs by downloading the base snapshot and
replaying deltas; this happens automatically when no local manifest is present.
Deletions are **never destructive** remotely: `--reconcile` only stops tracking a
removed path in the manifest — the bucket keeps every previously uploaded
version.
incpush also **never modifies, moves, or deletes anything in your source folder**
— it only reads files and writes its own `.incpush/` state directory.
### Note on non-ASCII filenames
Object keys go through `object_store`, which percent-encodes non-ASCII bytes in
the remote key (e.g. `café.txt` is stored as `caf%C3%A9.txt`). ASCII names,
including spaces, are stored verbatim. This is deterministic and reversible, and
incpush keys its own manifest on the real local path, so diffing and restore
through incpush stay faithful — but the raw object name in the bucket is encoded.
`--adopt` is unaffected: it percent-decodes remote keys before matching, so a
unicode-named file already in the bucket (whether stored raw or encoded) is still
recognized and not re-uploaded.
## Development
```sh
cargo test # unit + end-to-end tests (the e2e suite runs against a file:// backend)
cargo build --release
```
## License
MIT