Skip to main content

Crate rcp_tools_rcp

Crate rcp_tools_rcp 

Source
Expand description

Fast file operations tools - rcp (copy) and rcpd (remote copy daemon)

This crate provides high-performance file copying tools that are significantly faster than traditional tools like cp when dealing with large numbers of files.

§Tools

§rcp - File Copy Tool

The rcp tool provides both local and remote file copying with exceptional performance.

§Local Copying

Basic local file copying, similar to cp but optimized for large filesets:

# Basic copy
rcp /source/path /dest/path --progress --summary

# Copy with metadata preservation
rcp /source /dest --preserve --progress

# Overwrite existing files
rcp /source /dest --overwrite --progress

§Remote Copying

Copy files between remote hosts using host:/path syntax (similar to scp):

# Copy from remote to local
rcp user@host:/remote/path /local/path --progress

# Copy from local to remote
rcp /local/path user@host:/remote/path --progress --preserve

# Copy between two remote hosts
rcp user@host1:/path1 user@host2:/path2 --progress --summary

When using remote paths, rcp automatically:

  • Starts rcpd daemons on remote hosts via SSH
  • Transfers data directly between source and destination (not through the master)
  • Resolves automatic file concurrency on the source rcpd and has the destination adopt it

Requirements for remote copying:

  • SSH access to remote hosts (uses your SSH config and keys)
  • rcpd binary available in the same directory as rcp on remote hosts

§rcpd - Remote Copy Daemon

The rcpd daemon is automatically launched by rcp when using remote paths. It should generally not be invoked directly by users.

The daemon operates in two modes:

  • Source mode: Reads files from the source host
  • Destination mode: Writes files to the destination host

§Key Features

§Performance

  • Parallel operations: Uses async I/O and worker threads for maximum throughput
  • Optimized for large filesets: Much faster than cp when dealing with many files
  • Direct transfer: Data flows directly between source and destination hosts

§Copy Semantics

Unlike cp, rcp uses unambiguous path semantics:

  • Without trailing slash: Path is treated as the final destination name
    • rcp A/B C/D → creates C/D (fails if exists)
  • With trailing slash: Path is treated as a directory to copy into
    • rcp A/B C/D/ → creates C/D/B (fails if exists)

Use --overwrite to allow overwriting existing destinations.

§Metadata Preservation

  • --preserve: Preserve owner, group, mode and timestamps (deprecated; use --preserve-settings=all). Does NOT preserve POSIX ACLs — detecting one costs a syscall on every entry, so ACLs are opt-in.
  • --preserve-settings: Fine-grained control over what to preserve, including all+acl and the per-type acl attribute for POSIX ACLs

§Error Handling

  • By default: Log errors and continue processing
  • --fail-early: Stop on first error

§Progress & Logging

  • --progress: Show progress (auto-detects terminal type)
  • --summary: Print summary statistics at the end
  • -v/-vv/-vvv: Control log verbosity (INFO/DEBUG/TRACE)
  • Progress goes to stderr, logs to stdout (allows piping logs while viewing progress)

§Throttling & Resource Control

  • --ops-throttle: Limit operations per second
  • --iops-throttle: Limit I/O operations per second
  • --max-files-in-flight: Cap applicable file-like work. Local automatic defaults resolve on the initiating host; remote defaults resolve on the source rcpd and are adopted by the destination (each with a floor of 4). Use unlimited to remove the user ceiling
  • --max-workers: Control number of worker threads

§Remote Copy Configuration

  • --port-ranges: Restrict TCP data ports to specific ranges (e.g., “8000-8999”)
  • --remote-copy-conn-timeout-sec: Positive timeout for bounded remote bootstrap stages and TCP connections (default: 15; 60 with --auto-deploy-rcpd)
  • --remote-keepalive-sec: Liveness budget for every rcp TCP connection in seconds, 0 disables (default: 120). Keepalive on every connection; TCP_USER_TIMEOUT on control connections only

§Architecture

§Local Copy Architecture

Local copying uses async I/O with multiple worker threads to maximize throughput when dealing with large numbers of files.

§Remote Copy Architecture

Remote copying uses a three-node architecture:

Master (rcp)
├── SSH → Source Host (rcpd in source mode)
│   ├── TCP Server ← Master (control + tracing)
│   └── TCP Server ← Destination (control + data ports)
└── SSH → Destination Host (rcpd in destination mode)
    ├── TCP Server ← Master (control + tracing)
    └── TCP Client → Source (control + data)

Connection flow:

  1. Master starts rcpd processes on both hosts via SSH
  2. Master connects to each rcpd’s TCP listener (control + tracing connections; the listener address is read from the rcpd’s stderr over SSH)
  3. Source rcpd starts TCP listeners and sends its addresses to Master
  4. Master forwards the addresses to Destination rcpd
  5. Destination connects directly to Source’s control and data ports
  6. Data flows directly from Source to Destination (not through Master)

This architecture ensures efficient data transfer while allowing the Master to coordinate the operation and monitor progress.

For detailed network connectivity and troubleshooting information, see the docs/remote_copy.md file in the repository.

§Examples

§Local Copy Examples

# Basic copy with progress
rcp /source /dest --progress --summary

# Copy preserving all metadata
rcp /source /dest --preserve --progress

# Copy multiple sources into a directory
rcp file1 file2 dir3 /dest/ --progress

§Remote Copy Examples

# Copy from remote host to local
rcp server:/data/files /local/backup --progress --preserve

# Copy to remote host
rcp /local/data server:/backup/ --progress --summary

# Copy between remote hosts with custom port ranges
rcp host1:/path1 host2:/path2 --port-ranges "8000-8999" --progress

§Library Usage

This crate also provides library functions for integrating remote copy functionality into other Rust applications. See the module documentation for details.

Modules§

destination
directory_tracker
Tracks directory completion state during remote copy operations.
path
source