rust-parallel-0.1.21 is not a library.

rust-parallel

Command-line utility to execute commands in parallel and aggregate their output.

Something like a simple rust version of GNU Parallel.

Being written in asynchronous rust it is quite fast - see benchmarks.

Goals:

Use only safe rust.
Use only asynchronous operations supported by tokio, do not use any blocking operations.
Support arbitrarily large number of input lines, avoid O(number of input lines) memory usage. In support of this:
- tokio::sync::Semaphore is used carefully to limit the number of commands that run concurrently. Do not spawn tasks for all input lines immediately to limit memory usage.
Support running commands on local machine only, not on remote machines.

Tech Stack:

anyhow used for application error handling to propogate and format fatal errors.
clap command line argument parser.
tokio asynchronous runtime for rust. From tokio this app uses:
- async / await functions (aka coroutines)
- Singleton CommandLineArgs instance using tokio::sync::OnceCell.
- Asynchronous command execution using tokio::process::Command
- tokio::sync::Semaphore used to limit number of commands that run concurrently.
  - Life would be a bit easier if acquire_many took a usize parameter: https://github.com/tokio-rs/tokio/issues/4446
- tokio::sync::Mutex used to protect access to stdout/stderr to prevent interleaved command output.
tracing used for debug and warning logs.

Installation:

Install Rust
Install the latest version of this app from crates.io:

$ cargo install rust-parallel

Usage:

$ rust-parallel -h

Run commands in parallel

Usage: rust-parallel [OPTIONS] [INPUTS]...

Arguments:
  [INPUTS]...  Input file or - for stdin.  Defaults to stdin if no inputs are specified

Options:
  -j, --jobs <JOBS>    Maximum number of commands to run in parallel, defauts to num cpus [default: 12]
  -s, --shell-enabled  Use /bin/sh -c shell to run commands
  -h, --help           Print help information
  -V, --version        Print version information

Demos:

Small demo of 5 echo commands. With -j5 all 5 commands are run in parallel. With -j1 commands are run sequentially:

$ cat >./test <<EOL
# input can contain comment lines (starting with #) and blank lines too

echo hi
echo there
echo how
echo are
echo you
EOL

$ cat test | rust-parallel -j5
are
hi
there
how
you

cat test | rust-parallel -j1
hi
there
how
are
you

Using awk to form commands:

$ head -100 /usr/share/dict/words| awk '{printf "md5 -s %s\n", $1}' | rust-parallel
MD5 ("Abba") = 5fa1e1f6e07a6fea3f2bb098e90a8de2
MD5 ("abaxial") = ac3a53971d52d9ce3277eadf03f13a5e
MD5 ("abaze") = 0b08c52aa63d947b6a5601ee975bc3a4
MD5 ("abaxile") = 21f5fc27d7d34117596e41d8c001087e
MD5 ("abbacomes") = 76640eb0c929bc97d016731bfbe9a4f8
MD5 ("abbacy") = 08aeac72800adc98d2aba540b6195921
MD5 ("Abbadide") = 7add1d6f008790fa6783bc8798d8c803
MD5 ("abb") = ea01e5fd8e4d8832825acdd20eac5104

Using as part of a shell pipeline. stdout and stderr from each command run are copied to stdout/stderr of the rust-parallel process.

$ head -100 /usr/share/dict/words| awk '{printf "md5 -s %s\n", $1}' | rust-parallel | grep -i abba
MD5 ("Abba") = 5fa1e1f6e07a6fea3f2bb098e90a8de2
MD5 ("abbacomes") = 76640eb0c929bc97d016731bfbe9a4f8
MD5 ("abbacy") = 08aeac72800adc98d2aba540b6195921
MD5 ("Abbadide") = 7add1d6f008790fa6783bc8798d8c803

Using input file. Multiple inputs can be specified, - means stdin:

$ cat >./test1 <<EOL
echo hi
echo there
echo how
EOL

$ cat >./test2 <<EOL
echo are
echo you
EOL

$ cat test2 | rust-parallel test1 -
there
how
hi
are
you

With debug logs enabled: