quick-file-transfer 0.8.0

qft

Purpose

Note! This is a work in progress and is in no means production ready

Transfer files as quickly, safely, and painlessly as possible on a local network.

This readme is also available as a book.

qft optimizes for a scenario where embedded systems regularly transfer large files across a local network, such as a continuous integration pipeline where firmware (e.g. Rauc) can take significant time to transfer with tools such as rsync, scp, or netcat.

To accomplish this, qft acts as a server/client that transfers data over TCP. It is very similar to how netcat can be used to transfer files, but qft focuses solely on transferring files, and comes with a variety of customization options such as compression/decompression, memory mapping, preallocation options and more. TCP is chosen for reliable data transfer, and no authentication or encryption is layered on top to reduce the overhead, addressing remote targets by mDNS is also supported.

If you are worried about a man-in-the-middle, you can simply check your data on the receiving end before continuing. There should be no additional security concerns (if you disagree, please create an issue highlighting the concern).

Features

• Send files via TCP by specifying either IP or mDNS/DNS-SD hostname
• Evaluate supported compression formats on your input data
• Discover, resolve, and/or register mDNS/DNS-SD services
• SCP like transfers qft ssh FILES... <user>@<host>:<path>. Where auth occurs via SSH but transfer is bare bone TCP.
• Shell completions for bash, elvish, fish, powershell, and zsh.

Usage

$ qft -h
Usage: qft [OPTIONS] [COMMAND]

Commands:
  listen                Run in Listen (server) mode
  send                  Run in Send (client) mode
  mdns                  Use mDNS utilities
  evaluate-compression  Evaluate which compression works best for file content
  get-free-port         Get a free port from the host OS. Optionally specify on which IP or a port range to scan for a free port
  help                  Print this message or the help of the given subcommand(s)

Options:
  -v, --verbose...           Pass many times for more log output
  -q, --quiet                Silence all log output, this will lead to better performance [env: QFT_QUIET=]
      --color=<WHEN>         [default: auto] [possible values: auto, always, never]
      --completions <SHELL>  Generate completion scripts for the specified shell. Note: The completion script is printed to stdout [possible values: bash, elvish, fish, powershell, zsh]
  -h, --help                 Print help (see more with '--help')
  -V, --version              Print version

Examples

File transfer

In a CI script using key based SSH auth, it looks very similar to SCP.

Both hosts need qft installed!

Host #1

qft ssh file.data foo@bar.local:/tmp/

CI script with no SSH auth

Something like a Raspberry Pi could orchestrate the testing of an embedded system, and might use a script like this to transfer a firmware upgrade bundle.

#!/usr/bin/env bash
set -eu
REMOTE_HOSTNAME="foo.local."
FIRMWARE="fw.raucb"
qft ssh ${FIRMWARE} root@${REMOTE_HOSTNAME}:/
ssh root@${REMOTE_HOSTNAME} -t "rauc install /${FIRMWARE}"
...

It is also possible to ad-hoc register a service with qft mdns register AND run the qft listen side-by-side and then send to the listening process by addressing the registered hostname from a remote host.

Evaluate compression

Evaluate which compression works best for file content

Usage: qft evaluate-compression [OPTIONS] --input-file <INPUT_FILE>

Options:
  -i, --input-file <INPUT_FILE>
      --omit [<OMIT>...]                List of compression formats to omit from evaluation [possible values: bzip2, gzip, lz4, xz]
      --omit-levels [<OMIT_LEVELS>...]  List of compression levels to omit from evaluation
  -v, --verbose...                      Pass many times for more log output
  -q, --quiet                           Silence all output [env: QFT_QUIET=]
      --color=<WHEN>                    [default: auto] [possible values: auto, always, never]
  -h, --help                            Print help (see more with '--help')

Evaluate compression of Cargo.lock. Omit gzip and most compression levels to make this example brief.

qft evaluate-compression --input-file Cargo.lock --omit gzip --omit-levels 0 2 3 4 5 6 7 8

Example output:

INFO Omitting:   Gzip
INFO Evaluating: Bzip2 Lz4 Xz
INFO Omitting compression levels (where applicable): 0 2 3 4 5 6 7 8
INFO Buffered reading 34338 B contents in 15.728µs
INFO Lz4
╭────────────────────┬─────────────────────╮
│ Compression Ratio  ┆        2.42:1       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Encode/decode time ┆   83.29µs/32.53µs   │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compressed Size    ┆ 13.83 KiB [14163 B] │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ % of Original      ┆        41.25%       │
╰────────────────────┴─────────────────────╯
INFO Bzip2
╭────────────────────┬───────────────────╮
│ Compression level  ┆         1         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compression Ratio  ┆       4.56:1      │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Encode/decode time ┆  2.36ms/686.34µs  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compressed Size    ┆ 7.36 KiB [7533 B] │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ % of Original      ┆       21.94%      │
╰────────────────────┴───────────────────╯
INFO Bzip2
╭────────────────────┬───────────────────╮
│ Compression level  ┆         9         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compression Ratio  ┆       4.56:1      │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Encode/decode time ┆  2.73ms/771.13µs  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compressed Size    ┆ 7.36 KiB [7533 B] │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ % of Original      ┆       21.94%      │
╰────────────────────┴───────────────────╯
INFO Xz
╭────────────────────┬───────────────────╮
│ Compression level  ┆         1         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compression Ratio  ┆       3.73:1      │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Encode/decode time ┆  2.34ms/522.16µs  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compressed Size    ┆ 9.00 KiB [9216 B] │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ % of Original      ┆       26.84%      │
╰────────────────────┴───────────────────╯
INFO Xz
╭────────────────────┬───────────────────╮
│ Compression level  ┆         9         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compression Ratio  ┆       4.30:1      │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Encode/decode time ┆  9.04ms/523.23µs  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compressed Size    ┆ 7.80 KiB [7984 B] │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ % of Original      ┆       23.25%      │
╰────────────────────┴───────────────────╯
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

╭────────────────────┬───────────────────┬───────────────────────┬─────────────────────────╮
│                    ┆ Best Ratio        ┆ Best Compression Time ┆ Best Decompression Time │
╞════════════════════╪═══════════════════╪═══════════════════════╪═════════════════════════╡
│ Format             ┆ Bzip2             ┆ Lz4                   ┆ Lz4                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compression level  ┆ 1                 ┆ -                     ┆ -                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compression Ratio  ┆ 4.56:1            ┆ 2.42:1                ┆ 2.42:1                  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Encode/decode time ┆ 2.36ms/686.34µs   ┆ 83.29µs/32.53µs       ┆ 83.29µs/32.53µs         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Compressed Size    ┆ 7.36 KiB [7533 B] ┆ 13.83 KiB [14163 B]   ┆ 13.83 KiB [14163 B]     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ % of Original      ┆ 21.94%            ┆ 41.25%                ┆ 41.25%                  │
╰────────────────────┴───────────────────┴───────────────────────┴─────────────────────────╯

==> Short summary
Best Compression Ratio:   Bzip2[1] Compression/Decompression:     2.36ms/  686.34µs   4.56:1 (21.94% of original)
Best Compression Time:    Lz4      Compression/Decompression:    83.29µs/   32.53µs   2.42:1 (41.25% of original)
Best Decompression Time:  Lz4      Compression/Decompression:    83.29µs/   32.53µs   2.42:1 (41.25% of original)

mDNS utilities

The purpose of the built-in mDNS/DNS-SD utilities are solely for easy network setup/testing/debugging, therefor they are generally more verbose and have much slower (but more complete) defaults than e.g. avahi does.

Discover services

qft mdns discover --service-label googlecast --service-protocol tcp

Example Output

INFO Browsing for _googlecast._tcp.local.
INFO Resolved a new service: SERVICE_NAME._googlecast._tcp.local.
INFO Discovered 1 service!
Hostname:  SERVICE_NAME.local.
Type Name: _googlecast._tcp.local.
Full Name: SERVICE_NAME._googlecast._tcp.local.
IP(s): fe80::d912:463a:8c88:deca
       192.168.121.21

Resolve mDNS hostname

Resolves hostname IP(s), all of the following forms are valid.

qft mdns resolve foo
qft mdns resolve foo.local
qft mdns resolve foo.local.

Example output

INFO Resolving address for foo.local.
Hostname:  foo.local.
IP(s): fe80::d912:463a:8c88:deca
       192.168.121.21

Register mDNS service (for testing or transferring by addressing the registered hostname)

qft mdns register --hostname foo-name --service-label bar-label --service-protocol tcp --keep-alive-ms 123456

INFO Registering:
    Hostname:  foo-name.local.
    Type:      _bar-label._tcp.local.
    Full Name: test_inst._bar-label._tcp.local.

INFO Keeping alive for: 123.456s

You can the find it using the qft mdns subcommands or e.g. with avahi:

avahi-resolve --name foo-name.local
# foo-name.local  172.17.0.1

But that only outputs the first received address. Using qft mdns resolve will output all the associated IPs. If you need speed use the --short-circuit flag to return as soon as the first IP associated with the hostname is resolved e.g.

qft mdns resolve foo-name[.local.] --short-circuit

Supported compression formats

Install

Build from source (preferred if you have the Rust toolchain installed).

cargo install quick-file-transfer

Prebuilt binaries

curl --proto '=https' --tlsv1.2 -sSf https://crambl.github.io/quick-file-transfer/install.sh | bash -s -- --to <DEST>

Comparison/Benchmarks

Benchmarks are done with hyperfine default settings.

Using a 7.2 MiB JSON-file (not prettified) I had nearby with real data.

Targeting a Raspberry Pi Zero W that is connected with ethernet to a gigabit network.

netcat-like mode

The RPI-0W was running qft listen -p <PORT> --output 7mb.json and nc -l <PORT> > 7mb.json on repeat for the duration of the benchmark.

scp-like mode