Crate rcp_tools_rcp

Crate rcp_tools_rcp 

Source
Expand description

Fast file operations tools - rcp (copy) and rcpd (remote copy daemon)

This crate provides high-performance file copying tools that are significantly faster than traditional tools like cp when dealing with large numbers of files.

§Tools

§rcp - File Copy Tool

The rcp tool provides both local and remote file copying with exceptional performance.

§Local Copying

Basic local file copying, similar to cp but optimized for large filesets:

# Basic copy
rcp /source/path /dest/path --progress --summary

# Copy with metadata preservation
rcp /source /dest --preserve --progress

# Overwrite existing files
rcp /source /dest --overwrite --progress

§Remote Copying

Copy files between remote hosts using host:/path syntax (similar to scp):

# Copy from remote to local
rcp user@host:/remote/path /local/path --progress

# Copy from local to remote
rcp /local/path user@host:/remote/path --progress --preserve

# Copy between two remote hosts
rcp user@host1:/path1 user@host2:/path2 --progress --summary

When using remote paths, rcp automatically:

  • Starts rcpd daemons on remote hosts via SSH
  • Uses QUIC protocol for efficient data transfer
  • Transfers data directly between source and destination (not through the master)

Requirements for remote copying:

  • SSH access to remote hosts (uses your SSH config and keys)
  • rcpd binary available in the same directory as rcp on remote hosts

§rcpd - Remote Copy Daemon

The rcpd daemon is automatically launched by rcp when using remote paths. It should generally not be invoked directly by users.

The daemon operates in two modes:

  • Source mode: Reads files from the source host
  • Destination mode: Writes files to the destination host

§Key Features

§Performance

  • Parallel operations: Uses async I/O and worker threads for maximum throughput
  • Optimized for large filesets: Much faster than cp when dealing with many files
  • QUIC protocol: Efficient network transport for remote operations

§Copy Semantics

Unlike cp, rcp uses unambiguous path semantics:

  • Without trailing slash: Path is treated as the final destination name
    • rcp A/B C/D → creates C/D (fails if exists)
  • With trailing slash: Path is treated as a directory to copy into
    • rcp A/B C/D/ → creates C/D/B (fails if exists)

Use --overwrite to allow overwriting existing destinations.

§Metadata Preservation

  • --preserve: Preserve all metadata (owner, group, mode, timestamps)
  • --preserve-settings: Fine-grained control over what to preserve

§Error Handling

  • By default: Log errors and continue processing
  • --fail-early: Stop on first error

§Progress & Logging

  • --progress: Show progress (auto-detects terminal type)
  • --summary: Print summary statistics at the end
  • -v/-vv/-vvv: Control log verbosity (INFO/DEBUG/TRACE)
  • Progress goes to stderr, logs to stdout (allows piping logs while viewing progress)

§Throttling & Resource Control

  • --ops-throttle: Limit operations per second
  • --iops-throttle: Limit I/O operations per second
  • --max-open-files: Control maximum open file descriptors
  • --max-workers: Control number of worker threads

§Remote Copy Configuration

  • --quic-port-ranges: Restrict QUIC to specific port ranges (e.g., “8000-8999”)
  • --remote-copy-conn-timeout-sec: Connection timeout in seconds (default: 15)

§Architecture

§Local Copy Architecture

Local copying uses async I/O with multiple worker threads to maximize throughput when dealing with large numbers of files.

§Remote Copy Architecture

Remote copying uses a three-node architecture:

Master (rcp)
├── SSH → Source Host (rcpd in source mode)
│   └── QUIC → Master
│   └── QUIC Server (waits for Destination)
└── SSH → Destination Host (rcpd in destination mode)
    └── QUIC → Master
    └── QUIC Client → Source

Connection flow:

  1. Master starts rcpd processes on both hosts via SSH
  2. Both rcpd processes connect back to Master via QUIC
  3. Source rcpd starts a QUIC server and sends its address to Master
  4. Master forwards the address to Destination rcpd
  5. Destination connects directly to Source
  6. Data flows directly from Source to Destination (not through Master)

This architecture ensures efficient data transfer while allowing the Master to coordinate the operation and monitor progress.

For detailed network connectivity and troubleshooting information, see the docs/network_connectivity.md file in the repository.

§Examples

§Local Copy Examples

# Basic copy with progress
rcp /source /dest --progress --summary

# Copy preserving all metadata
rcp /source /dest --preserve --progress

# Copy multiple sources into a directory
rcp file1 file2 dir3 /dest/ --progress

§Remote Copy Examples

# Copy from remote host to local
rcp server:/data/files /local/backup --progress --preserve

# Copy to remote host
rcp /local/data server:/backup/ --progress --summary

# Copy between remote hosts with custom port ranges
rcp host1:/path1 host2:/path2 --quic-port-ranges "8000-8999" --progress

§Library Usage

This crate also provides library functions for integrating remote copy functionality into other Rust applications. See the module documentation for details.

Modules§

destination
directory_tracker
path
source