Skip to main content

Crate remote

Crate remote 

Source
Expand description

Remote copy protocol and networking for distributed file operations

This crate provides the networking layer and protocol definitions for remote file copying in the RCP tools suite. It enables efficient distributed copying between remote hosts using SSH for orchestration and TCP for high-throughput data transfer.

§Overview

The remote copy system uses a three-node architecture:

Master (rcp)
├── SSH → Source Host (rcpd)
│   └── TCP → Master (control)
│   └── TCP Server (accepts data connections from Destination)
└── SSH → Destination Host (rcpd)
    └── TCP → Master (control)
    └── TCP Client → Source (data transfer)

§Connection Flow

  1. Initialization: Master starts rcpd processes on source and destination via SSH
  2. Control Connections: Both rcpd processes connect back to Master via TCP
  3. Address Exchange: Source starts TCP listeners and sends addresses to Master
  4. Direct Connection: Master forwards addresses to Destination, which connects to Source
  5. Data Transfer: Files flow directly from Source to Destination (not through Master)

This design ensures efficient data transfer while allowing the Master to coordinate operations and monitor progress.

§Key Components

§SSH Session Management

The SshSession type represents an SSH connection to a remote host and is used to:

  • Launch rcpd daemons on remote hosts
  • Configure connection parameters (user, host, port)

§TCP Networking

TCP provides high-throughput bulk data transfer with:

  • Connection pooling for parallel file transfers
  • Configurable buffer sizes for different network profiles
  • Length-delimited message framing for control messages

Key functions:

§Port Range Configuration

The port_ranges module allows restricting TCP to specific port ranges, useful for firewall-restricted environments:

let config = remote::TcpConfig::default().with_port_ranges("8000-8999");
let listener = remote::create_tcp_control_listener(&config, None).await?;

§Protocol Messages

The protocol module defines the message types exchanged between nodes:

  • MasterHello - Master → rcpd configuration
  • SourceMasterHello - Source → Master address information
  • RcpdResult - rcpd → Master operation results
  • TracingHello - rcpd → Master tracing initialization

§Stream Communication

The streams module provides high-level abstractions over TCP streams:

§Remote Tracing

The tracelog module enables distributed logging and progress tracking:

  • Forward tracing events from remote rcpd processes to Master
  • Aggregate progress information across multiple remote operations
  • Display unified progress for distributed operations

§Security Model

The remote copy system provides multiple security layers:

  • SSH Authentication: All remote operations require SSH authentication to spawn rcpd
  • TLS Encryption: All TCP connections encrypted with TLS 1.3 by default
  • Certificate Pinning: Self-signed certificates with fingerprint verification
  • Mutual Authentication: Source↔Destination connections use mutual TLS

How it works:

  1. Each rcpd generates an ephemeral self-signed certificate
  2. rcpd outputs its certificate fingerprint to SSH stdout (trusted channel)
  3. Master connects to rcpd as TLS client, verifies fingerprint
  4. Master distributes fingerprints to enable Source↔Destination mutual auth

Use --no-encryption to disable TLS for trusted networks where performance is critical.

§Network Troubleshooting

Common failure scenarios:

  • SSH Connection Fails: Host unreachable or authentication failure
  • rcpd Cannot Connect to Master: Firewall blocks TCP ports
  • Destination Cannot Connect to Source: Use --port-ranges to specify allowed ports

§Module Organization

  • port_ranges - Port range parsing and socket binding
  • protocol - Protocol message definitions and serialization
  • streams - TCP stream wrappers with typed message passing
  • tls - TLS certificate generation and configuration
  • tracelog - Remote tracing and progress aggregation

Modules§

deploy
Binary deployment for rcpd
port_ranges
protocol
Remote copy protocol definitions for source-destination communication.
streams
tls
TLS support for encrypted and authenticated connections.
tracelog

Structs§

RcpdConnectionInfo
Connection info received from rcpd after it starts listening.
RcpdProcess
Result of starting an rcpd process.
SshSession
TcpConfig
Configuration for TCP connections

Enums§

NetworkProfile
Network profile for TCP configuration tuning

Constants§

DATACENTER_REMOTE_COPY_BUFFER_SIZE
Datacenter profile: buffer size for remote copy operations (16 MiB)
DEFAULT_PENDING_WRITES_MULTIPLIER
Default multiplier for pending writes (4× max_connections)
INTERNET_REMOTE_COPY_BUFFER_SIZE
Internet profile: buffer size for remote copy operations (2 MiB)

Functions§

configure_tcp_buffers
Configure TCP socket buffer sizes for high throughput
connect_tcp_control
Connect to a TCP control server with timeout
create_tcp_control_listener
Create a TCP listener for control connections
create_tcp_data_listener
Create a TCP listener for data connections (file transfers)
get_random_server_name
Generate a random server name for identifying connections
get_remote_home
Validate and retrieve HOME directory on remote host
get_remote_home_for_session
get_tcp_listener_addr
Get the external address of a TCP listener
is_localhost
checks if a host string refers to the local machine. returns true for localhost, 127.0.0.1, ::1, [::1], or the actual hostname
start_rcpd
wait_for_rcpd_process