pxs

pxs (Parallel X-Sync) is a file synchronization tool written in Rust for the same broad job as rsync: move data trees efficiently and refresh existing copies with as little work as possible.

The name is intentionally short for CLI use: pxs stands for Parallel X-Sync.

pxs focuses on modern large-data sync workloads, such as repeated refreshes of large PostgreSQL PGDATA directories, VM images, and other datasets with many unchanged files or large files that are updated in place.

rsync remains the reference point in this space. pxs is not a drop-in replacement for it. The goal is narrower: use Rust performance, parallelism, concurrency, fixed-block delta sync, and high-throughput transport to speed up data synchronization for workloads where those choices help.

Key Features

• Multi-threaded Engine: Parallelizes file walking, block-level hashing, and I/O operations.
• Fixed-Block Synchronization: Uses 128KB chunks and XxHash64 for ultra-fast delta analysis.
• High-Throughput TCP Transport: Uses a compact binary protocol with rkyv serialization over raw TCP.
• Auto-SSH Mode: Seamlessly tunnels through SSH for secure transfers without manual port forwarding.
• Pull Mode: Supports both pushing to and pulling from remote servers.
• Staged Atomic Writes: Preserves an existing destination until the replacement file is fully written and ready to commit.
• Smart Skipping: Automatically skips unchanged files based on size and modification time.

Installation

Install from crates.io:

cargo install pxs

Build from source:

cargo build --release

The binary will be available at ./target/release/pxs.

[!IMPORTANT] For Network or SSH synchronization, pxs must be installed and available in the $PATH on both the source and destination servers.

[!NOTE] Clock Synchronization: When using mtime-based skip detection (the default without --checksum), ensure source and destination systems have synchronized clocks (e.g., via NTP). Clock skew can cause files to be incorrectly skipped or unnecessarily re-synced. Use --checksum to force content-based comparison if clock sync is not guaranteed.

Platform Support

pxs currently targets Unix-like systems only:

• Linux
• macOS
• BSD

Windows is not supported.

For network and --stdio transports, pxs uses normalized relative POSIX paths in the protocol. Incoming paths are rejected if they are absolute or contain . / .. traversal components. Paths containing \ are also rejected by the protocol, so filenames with backslashes are not supported for remote sync.

How It Works

Local Synchronization

flowchart LR
    SRC[Source path] --> WALK[Parallel file walker]
    WALK --> HASH[Parallel block hasher]
    HASH --> COMPARE[Block comparator]
    COMPARE -->|Changed blocks only| WRITE[Parallel block writer]
    WRITE --> DST[Destination path]

Mermaid source: docs/diagrams/local-sync.mmd Fallback image: docs/diagrams/local-sync.svg

Network Synchronization (Direct TCP)

sequenceDiagram
    participant S as Sender
    participant R as Receiver

    S->>R: Handshake
    R->>S: Handshake ACK

    loop For each file
        S->>R: SyncFile(path, metadata, size)
        alt Destination can delta sync
            R->>S: RequestHashes
            S->>R: BlockHashes
            R->>S: RequestBlocks(changed indexes)
            S->>R: ApplyBlocks(delta data)
        else Full copy required
            R->>S: RequestFullCopy
            S->>R: ApplyBlocks(all data)
        end
        S->>R: ApplyMetadata
        R->>S: MetadataApplied
    end

Mermaid source: docs/diagrams/direct-tcp.mmd Fallback image: docs/diagrams/direct-tcp.svg

SSH Synchronization (Auto-Tunnel)

flowchart LR
    CLI[Local pxs CLI] -->|starts| SSH[SSH process]
    CLI <-->|pxs protocol over stdio| SSH
    SSH <-->|encrypted transport| REMOTE[Remote pxs --stdio]
    REMOTE --> DST[Destination path]

Mermaid source: docs/diagrams/ssh-flow.mmd Fallback image: docs/diagrams/ssh-flow.svg

Delta Sync Algorithm

flowchart TD
    START([Start sync]) --> EXISTS{Destination exists?}
    EXISTS -->|No| FULL[Full copy]
    EXISTS -->|Yes| SIZE{Size matches?}
    SIZE -->|No| THRESH{Below threshold?}
    THRESH -->|Yes| FULL
    THRESH -->|No| DELTA[Delta sync]
    SIZE -->|Yes| MTIME{mtime matches and no checksum?}
    MTIME -->|Yes| SKIP[Skip file]
    MTIME -->|No| DELTA
    DELTA --> HASH[Hash source and destination blocks]
    HASH --> DIFF[Compare block hashes]
    DIFF --> APPLY[Transfer changed blocks only]
    FULL --> META[Apply metadata]
    APPLY --> META
    SKIP --> DONE([Done])
    META --> DONE

Mermaid source: docs/diagrams/delta-sync.mmd Fallback image: docs/diagrams/delta-sync.svg

Usage

sync

Use this when both source and destination are local paths on the same machine. sync is the default local data-mover: it compares an existing destination and only rewrites changed blocks when delta sync is worthwhile.

Typical use:

• local file or directory refresh
• repeated sync of a local PGDATA copy
• local copy where you still want pxs block-level behavior instead of cp

Examples:

# Synchronize a single file
pxs sync file.bin backup.bin

# Synchronize a directory
pxs sync /path/to/source_dir /path/to/dest_dir

# Force checksum-based verification
pxs sync ./dataset.bin /mnt/backup/dataset.bin --checksum

# Flush file data to disk before completion
pxs sync ./dataset.bin /mnt/backup/dataset.bin --fsync

push

Use this when the data starts on the local machine and you want to send it somewhere else. push pairs with listen for raw TCP transfers, or it can target an SSH endpoint directly.

Typical use:

• send local data to a remote receiver over raw TCP
• push directly to a remote path over SSH
• benchmark sender-side transfer performance

Examples:

# Push one file to a raw TCP receiver
pxs push ./archive.tar 192.168.1.10:8080

# Push a directory tree to a raw TCP receiver
pxs push /var/lib/postgresql/data 192.168.1.10:8080

# Push one file over SSH
pxs push ./backup.tar.zst db2@example.net:/srv/backups/backup.tar.zst

# Push a directory tree over SSH
pxs push /var/lib/postgresql/data db2@example.net:/srv/replica/data

pull

Use this when the data should end up on the local machine. pull pairs with serve for raw TCP transfers, or it can fetch directly from an SSH endpoint.

Typical use:

• fetch data from a remote source into a local directory
• pull a remote snapshot or PGDATA tree over SSH
• run the receiving side locally while the remote side exposes data

Examples:

# Pull from a raw TCP serve endpoint
pxs pull 192.168.1.10:8080 ./snapshot.bin

# Pull one file over SSH
pxs pull db1@example.net:/srv/export/base.tar.zst ./base.tar.zst

# Pull a directory tree over SSH
pxs pull db1@example.net:/var/lib/postgresql/data /srv/restore/data

For raw TCP endpoints, source-side options such as --checksum, --threshold, and --ignore belong on serve. For SSH endpoints, pull can pass those options through to the remote helper.

listen

Use this when this machine should receive incoming push operations. listen owns the destination path and waits for another host to push data into it.

Typical use:

• prepare a destination host for an incoming raw TCP push
• expose a durable receiving endpoint with --fsync

Examples:

# Receive files into /srv/incoming
pxs listen 0.0.0.0:8080 /srv/incoming

# Receive into /new/data and fsync committed files
pxs listen 0.0.0.0:8080 /new/data --fsync

serve

Use this when this machine should expose a source tree for remote pull clients. serve is the mirror image of listen: it owns the source path and waits for another host to pull from it.

Typical use:

• serve a local snapshot over raw TCP
• keep source-side filtering or checksum policy on the source host

Examples:

# Serve one file for remote pull clients
pxs serve 0.0.0.0:8080 /srv/export/snapshot.bin

# Serve a directory tree with checksum verification enabled
pxs serve 0.0.0.0:8080 /srv/export/pgdata --checksum

Raw TCP Command Pairs

Use these pairings for direct TCP flows on trusted networks:

# Remote host receives an incoming push
pxs listen 0.0.0.0:8080 /srv/incoming
# Local host sends data to it
pxs push /var/lib/postgresql/data 192.168.1.10:8080

# Remote host exposes data for pull clients
pxs serve 0.0.0.0:8080 /srv/export/snapshot.bin
# Local host pulls it down
pxs pull 192.168.1.10:8080 ./snapshot.bin

SSH Command Pairs

Use these when you want pxs to manage the SSH tunnel automatically:

# Push local data to a remote path over SSH
pxs push my_file.bin user@remote-server:/path/to/dest/my_file.bin

# Pull remote data into a local path over SSH
pxs pull user@remote-server:/path/to/remote/file.bin ./local_file.bin

Manual SSH (using stdio pipe)

If you need custom SSH flags, you can still use the internal --stdio transport manually:

ssh user@remote-server "pxs --stdio --quiet --destination /path/to/new/data" < <(pxs push /path/to/old/data -)

Advanced Options

• --quiet (-q): Suppress all progress bars and status messages.
• --checksum (-c): Force a block-by-block hash comparison even if size/mtime match.
• --fsync (-f): Force fsync(2) after file writes. Slower, but safer for durability-sensitive copies.
• --ignore (-i): (Repeatable) Skip files/directories matching a glob pattern (e.g., -i "*.log").
• --exclude-from (-E): Read exclude patterns from a file (one pattern per line).
• --threshold (-t): (Default: 0.5) If the destination file is less than X% the size of the source, perform a full copy instead of hashing.
• --dry-run (-n): Show what would have been transferred without making any changes.
• --verbose (-v): Increase logging verbosity (use -vv for debug).

Progress Output & Quiet Mode

pxs provides a real-time, multi-threaded progress display for local and network synchronizations.

Aggregate and Individual Progress

For directory synchronizations, pxs shows:

1. Main Progress Bar (Top): An aggregate bar tracking the total bytes processed across the entire directory tree.
2. Worker Bars (Below): Individual progress bars for each large file currently being processed by a worker thread.

Throttled Bar Creation (Performance Optimization)

To ensure maximum performance and terminal readability, pxs uses a "smart" progress strategy:

• Small Files (< 1MB): Files smaller than 1MB are processed so quickly that creating a progress bar would cause excessive terminal flickering and CPU overhead. These files silently increment the Main Progress Bar without showing a dedicated sub-bar.
• Large Files (>= 1MB): Files 1MB or larger get a dedicated line in the terminal showing their specific transfer speed and completion percentage.
• Worker Limits: The number of concurrent file progress bars is limited by your hardware (CPU core count, capped at 64). This ensures that even when syncing millions of files, the terminal remains clean and the UI overhead stays negligible.

Quiet Mode

For use in cron jobs, scripts, or CI/CD pipelines, use the quiet flag to suppress all terminal output:

# Sync without any progress bars or status messages
pxs sync /src /dst --quiet
# or using the short flag
pxs sync /src /dst -q

Exclude Example

If you want to skip Postgres configuration files during a sync:

pxs sync /var/lib/postgresql/data /backup/data \
  --ignore "postmaster.opts" \
  --ignore "pg_hba.conf" \
  --ignore "postgresql.conf"

Or using a file:

echo "postmaster.pid" > excludes.txt
echo "*.log" >> excludes.txt
pxs sync /src /dst -E excludes.txt

How the Ignore Mechanism Works

pxs uses the same high-performance engine as ripgrep (the ignore crate) to filter files during the synchronization process.

Default Behavior (Full Clone)

By default, pxs is configured for Total Data Fidelity. It will NOT skip:

• Hidden files or directories (starting with .).
• Files listed in .gitignore.
• Global or local ignore files.

Using Patterns (Globs)

When you provide patterns via --ignore or --exclude-from, they are applied as overrides. Matching files are skipped entirely: they are not hashed, not counted in the total size, and not transferred.

Pattern	Effect
postmaster.pid	Ignores this specific file anywhere in the tree.
*.log	Ignores all files ending in .log.
temp/*	Ignores everything inside the top-level temp directory.
**/cache/*	Ignores everything inside any directory named cache at any depth.

Exclusion Pass-through (SSH)

When using Auto-SSH mode, your local ignore patterns are automatically sent to the remote server. This ensures that the receiver doesn't waste time looking at files you've already decided to skip.

Why pxs can be faster than rsync for some workloads

Feature	rsync	pxs
File hashing	Single-threaded	Parallel (all CPU cores)
Block comparison	Single-threaded	Parallel
Network transport	rsync protocol over remote shell or daemon	Raw TCP or SSH tunneled pxs protocol
Directory walking	Sequential	Parallel
Algorithm	Rolling hash	Fixed 128KB blocks

1. Parallelism and concurrency: pxs uses multiple CPU cores for hashing, comparison, file walking, and other hot-path work.
2. Algorithm choice: For workloads like database files, where data is usually modified in place rather than shifted, fixed-block delta sync can be cheaper than a rolling-hash approach.
3. Transport choice: On trusted high-speed networks, raw TCP avoids SSH overhead. When SSH is required, pxs still keeps its own transfer protocol and delta logic.

These advantages are workload-dependent. pxs shares rsync's goal of keeping data in sync, but it is aimed at repeated large-file and large-dataset refreshes on modern hardware rather than replacing rsync for every synchronization scenario.

Tests

The project includes a robust test suite for both local and network logic:

# Run all tests
cargo test

Podman end-to-end tests are also available:

# Direct TCP pull using serve/pull
./tests/podman/test_tcp_pull.sh

# SSH pull end-to-end
./tests/podman/test_ssh_pull.sh

# SSH push end-to-end
./tests/podman/test_ssh_push.sh

# SSH pull resume/truncation end-to-end
./tests/podman/test_ssh_pull_resume.sh

# Direct TCP push end-to-end
./tests/podman/test_tcp_push.sh

# Direct TCP directory/resume edge cases end-to-end
./tests/podman/test_tcp_directory_resume.sh

License

BSD-3-Clause