GNU Parallel

GNU Parallel lets us run shell commands in parallel.

# apt-get install parallel
$ mkdir -p ~/.parallel && touch ~/.parallel/will-cite

Parallel accepts a list of arguments, which we provide after ::::

$ parallel echo ::: alpha bravo charlie

If we supply multiple lists of arguments, parallel combines them:

$ parallel echo ::: a b ::: c d
a c
a d
b c
b d

The output order my vary.

The tool provides a number of replacement string options. The default string {} represents the input:

$ parallel echo {} ::: /tmp

(i.e. {} is replaced by a line of input, STDIN by default or else the arguments following :::.)

The replacement string {/} removes everything up to and including the last forward slash:

$ parallel echo {/} ::: /tmp/test.txt

If you want to return the path only, use the {//} string:

$ parallel echo {//} ::: /tmp/test.txt

The string {.} removes any filename extension:

$ parallel echo {.} ::: /tmp/test.txt

See the man page for more substitution and argument options (e.g. :::: to specify input read from files).

To run the same command multiple times for the same argument:

$ seq 5 | parallel -n0 echo "Hello!"

Test with --dry-run:

$ parallel --dry-run -k sleep {}\; echo {} ::: 5 2 1 4 3
sleep 5; echo 5
sleep 2; echo 2
sleep 1; echo 1
sleep 4; echo 4
sleep 3; echo 3

(-k forces the output order to match the arguments; I’m not certain that implies any guarantee about the order of execution.)