xcp 0.2.0

xcp is a (partial) clone of the Unix `cp` command with some more user-friendly feedback and some optimisations.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

# xcp: An extended cp

`xcp` is a (partial) clone of the Unix `cp` command. It is not intended as a
full replacement, but as a companion utility with some more user-friendly
feedback and some optimisations that make sense under certain tasks (see
below).

[![Crates.io](https://img.shields.io/crates/v/xcp.svg?colorA=777777)](https://crates.io/crates/xcp)

*Warning*: `xcp` is currently pre-alpha level software and almost certainly contains
bugs and unexpected or inconsistent behaviour. It probably shouldn't be used for
anything critical yet.

## Installation

### Arch Linux

[`xcp`](https://aur.archlinux.org/packages/xcp/) is available on the Arch Linux User Repository. If you use an AUR helper, you can execute a command such as this:
```
yay -S xcp
```

## Features and Anti-Features

### Features

* Displays a progress-bar, both for directory and single file copies. This can
  be disabled with `--no-progress`.
* On Linux it uses `copy_file_range` call to copy files. This is the most
  efficient method of file-copying under Linux; in particular it is
  filesystem-aware, and can massively speed-up copies on network mounts by
  performing the copy operations server-side. However, unlike `copy_file_range`
  sparse files are detected and handled appropriately.
* Non-Linux Unix-like OSs (OS X, *BSD) are supported via fall-back operation
  (although sparse-files are not supported).
* Optionally understands `.gitignore` files to limit the copied directories.
* Optimised for 'modern' systems (i.e. multiple cores, copious RAM, and
  solid-state disks, especially ones connected into the main system bus,
  e.g. M.2).
  
### (Possible) future features

* Optional aggressive parallelism for systems with parallel IO. Quick
  experiments on a modern laptop suggest there may be benefits to parallel
  copies on NVMe disks. This is obviously highly system-dependent.
* Conversion of files to sparse where appropriate, as with `cp`'s
  `--sparse=always` flag.
* Aggressive sparseness detection with `lseek`.

### Anti-Features

* On Linux it requires a kernel version of 4.5 and onwards.
* On non-Linux OSs sparse-files are not supported (although could be added is
  supported by the OS).
* Assumes a 'modern' system with lots of RAM and fast, solid-state disks. In
  particular it is likely to thrash on spinning disks as it attempts to gather
  metadata and perform copies at the same time.
* Currently missing a lot of `cp`'s features and flags, although these could be
  added.

## Performance

Benchmarks are mostly meaningless, but to check we're not introducing _too_ much
overhead for local copies, the following are results from a laptop with an NVMe
disk and in single-user mode. The target copy directory is a git checkout of the
Firefox codebase, having been recently gc'd (i.e. a single 4.1GB pack
file). `fstrim -va` is run before each test run to minimise SSD allocation
performance interference.

### Local copy

* Single 4.1GB file copy, with the kernel cache dropped each run:
    * `cp`: ~6.2s
    * `xcp`: ~4.2s
* Single 4.1GB file copy, warmed cache (3 runs each):
    * `cp`: ~1.85s
    * `xcp`: ~1.7s
* Directory copy, kernel cache dropped each run:
    * `cp`: ~48s
    * `xcp`: ~56s
* Directory copy, warmed cache (3 runs each):
    * `cp`: ~6.9s
    * `xcp`: ~7.4s

### NFS copy

`xcp` uses `copy_file_range`, which is filesystem aware. On NFSv4 this will result
in the copy occurring server-side rather than transferring across the network. For
large files this can be a significant win:

* Single 4.1GB file on NFSv4 mount
    * `cp`: 378s
    * `xcp`: ~37s