GNU Parallel | Cheatsheet

GNU Parallel is a powerful tool that allows for the concurrent execution of jobs, making efficient use of multiple CPU cores. It is widely used for various tasks, such as processing large data files and performing distributed computations across a network. This cheatsheet provides a comprehensive collection of practical examples to demonstrate the functionality of GNU Parallel.

Run multiple commands in parallel

echo -e "sleep 1\necho Hello" | parallel

Run commands from a file in parallel

parallel :::: commands.txt

Run commands from a script in parallel

parallel ./myscript.sh ::: arg1 arg2 arg3

Run commands with multiple arguments

parallel echo ::: A B C ::: 1 2 3

Run commands with multiple arguments from a file

parallel -a argfile.txt echo

Parallelize a shell loop

parallel 'echo processing file {}; mv {} {}.bak' ::: *.txt

Parallelize a find command

find . -name "*.txt" | parallel gzip {}

Parallelize a grep command

parallel grep pattern ::: file1 file2 file3

Control the number of jobs

parallel -j4 echo ::: A B C D E F G

Control the number of jobs based on CPU

parallel -j% echo ::: {1..100}

Set the maximum load

parallel --load 75% echo ::: {1..100}

Use a job slot with arguments

parallel -j2 'sleep {} && echo {}' ::: 1 2 3 4 5

Retrying Failed Jobs

parallel --retries 3 --timeout 200% faulty-command ::: inputs

Dry-run to check the commands

parallel --dry-run command ::: arg1 arg2

Batch Processing and Parallelism

Process groups of 10 jobs, count to 1000, and echo group parts

for i in {1..1000}; do
        echo $i
        if ((i % 10 == 0)); then
                echo "Group part reached"
        fi
done | parallel -j 10 'echo {}'

Run 10 parallel jobs and count to 10

NUM="10"
seq $NUM | parallel -n1 -P$NUM 'seq 10 | parallel -j10 echo {}'

Process groups of 4 jobs in parallel, count to 1000, and echo group parts

for i in {1..1000}; do
        echo $i
        if ((i % 5 == 0)); then
                echo "Group part reached"
        fi
done | parallel -j 4 'echo {}'

Handle a slow job example with 4 jobs in parallel

for i in {1..1000}; do
        echo $i
        if ((i % 5 == 0)); then
                echo "This is a slow job"
                sleep 1
        fi
done | parallel -j 4 'echo {}'

Advanced line grouping and parallel processing

input_file="input.txt"
lines_per_group=4

counter=0
group=""
while IFS= read -r line; do
  group+="$line"$'\n'
  counter=$((counter + 1))
  
  if ((counter % lines_per_group == 0)); then
    echo -n "$group" | parallel -j4 'echo {}'
    group=""
  fi
done <"$input_file"

# Check if there are any lines left at the end of the file
if [ -n "$group" ]; then
  echo -n "$group" | parallel -j4 'echo {}'
fi

Execute 4 parallel commands every second

for i in {1..1000}; do
  echo $i
  if ((i % 5 == 0)); then
    sleep 1
  fi
done | parallel -j 4 'echo {}'

Expand tab-separated variables in parallel

echo -e "Var1\tVar2\tVar3" | parallel --colsep '\t' 'echo {1} and {2} and {3}'

Expand line-separated variables in parallel

echo -e "Var1\nVar2\nVar3" | parallel 'echo {}'

Creating files in parallel

seq 100 | parallel 'touch file{}.txt'

Renaming files in parallel

ls *.txt | parallel 'mv {} {.}.bak'

Removing files in parallel

ls *.txt | parallel 'rm {}'

Fast and efficient file searching

find . -type f | parallel grep -l "search pattern"

Use GNU Parallel with rm -rf

find . -name "*.tmp" | parallel 'rm -rf {}'

Using the progress bar

seq 100 | parallel --progress 'sleep 0.1 && echo {}'

Displaying ETA (Estimated Time of Arrival)

seq 100 | parallel --eta 'sleep 0.1 && echo {}'

Pipe output to a file

seq 100 | parallel 'echo {}' > output.txt

Run command on multiple servers

echo -e "server1\nserver2\nserver3" | parallel --tag -j0 ssh {} hostname

Transfer and execute a script on multiple servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'scp myscript.sh {}:/tmp/; ssh {} bash /tmp/myscript.sh'

Run different commands on different servers

echo -e "server1:uname\nserver2:hostname\nserver3:date" | parallel --colsep ':' ssh {1} {2}

Run a command on each file in a directory on multiple servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "ls /path/to/directory | parallel grep pattern"'

Copy multiple files to multiple servers

parallel scp ::: file1 file2 file3 ::: server1:/path server2:/path server3:/path

Copy multiple directories to multiple servers

parallel scp -r ::: dir1 dir2 dir3 ::: server1:/path server2:/path server3:/path

Run a command on each line of file on multiple servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "cat /path/to/file | parallel command"'

Backup a directory to multiple servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "tar -czf /backup_dir/backup.tar.gz /path/to/directory"'

Update package on multiple servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag ssh {} sudo apt-get update

Running Commands in Parallel

Run multiple commands in parallel

echo -e "sleep 1\necho Hello" | parallel

Run commands from a file in parallel

parallel :::: commands.txt

Run commands from a script in parallel

parallel ./myscript.sh ::: arg1 arg2 arg3

Run commands with multiple arguments

parallel echo ::: A B C ::: 1 2 3

Run commands with multiple arguments from a file

parallel -a argfile.txt echo

Parallelize a shell loop

parallel 'echo processing file {}; mv {} {}.bak' ::: *.txt

Parallelize a find command

find . -name "*.txt" | parallel gzip {}

Parallelize a grep command

parallel grep pattern ::: file1 file2 file3

Controlling the Number of Jobs

parallel -j4 echo ::: A B C D E F G

Controlling the Number of Jobs Based on CPU

parallel -j% echo ::: {1..100}

Setting the Maximum Load

parallel --load 75% echo ::: {1..100}

Using a Job Slot with Arguments

parallel -j2 'sleep {} && echo {}' ::: 1 2 3 4 5

Retrying Failed Jobs

parallel --retries 3 --timeout 200% faulty-command ::: inputs

Dry-Run to Check the Commands

parallel --dry-run command ::: arg1 arg2

Batch Processing and Parallelism

Process Groups of Jobs and Echo Group Parts

for i in {1..1000}; do
        echo $i
        if ((i % 10 == 0)); then
                echo "Group part reached"
        fi
done | parallel -j 10 'echo {}'

Run Parallel Jobs within Parallel Jobs

NUM="10"
seq $NUM | parallel -n1 -P$NUM 'seq 10 | parallel -j10 echo {}'

Process Groups of Jobs, Echo Group Parts, and Control Parallelism

for i in {1..1000}; do
        echo $i
        if ((i % 5 == 0)); then
                echo "Group part reached"
        fi
done | parallel -j 4 'echo {}'

Handling Slow Jobs with Parallelism

for i in {1..1000}; do
        echo $i
        if ((i % 5 == 0)); then
                echo "This is a slow job"
                sleep 1
        fi
done | parallel -j 4 'echo {}'

Advanced Line Grouping and Parallel Processing

input_file="input.txt"
lines_per_group=4

counter=0
group=""
while IFS= read -r line; do
  group+="$line"$'\n'
  counter=$((counter + 1))
  
  if ((counter % lines_per_group == 0)); then
    echo -n "$group" | parallel -j4 'echo {}'
    group=""
  fi
done <"$input_file"

# Check if there are any lines left at the end of the file
if [ -n "$group" ]; then
  echo -n "$group" | parallel -j

4 'echo {}'
fi

Execute Parallel Commands with Delay

for i in {1..1000}; do
  echo $i
  if ((i % 5 == 0)); then
    sleep 1
  fi
done | parallel -j 4 'echo {}'

Expand Tab-Separated Variables in Parallel

echo -e "Var1\tVar2\tVar3" | parallel --colsep '\t' 'echo {1} and {2} and {3}'

Expand Line-Separated Variables in Parallel

echo -e "Var1\nVar2\nVar3" | parallel 'echo {}'

Creating Files in Parallel

seq 100 | parallel 'touch file{}.txt'

Renaming Files in Parallel

ls *.txt | parallel 'mv {} {.}.bak'

Removing Files in Parallel

ls *.txt | parallel 'rm {}'

Fast and Efficient File Searching

find . -type f | parallel grep -l "search pattern"

Using GNU Parallel with rm -rf

find . -name "*.tmp" | parallel 'rm -rf {}'

Running a Command on Multiple Servers

echo -e "server1\nserver2\nserver3" | parallel --tag -j0 ssh {} hostname

Transferring and Executing a Script on Multiple Servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'scp myscript.sh {}:/tmp/; ssh {} bash /tmp/myscript.sh'

Running Different Commands on Different Servers

echo -e "server1:uname\nserver2:hostname\nserver3:date" | parallel --colsep ':' ssh {1} {2}

Running a Command on Each File in a Directory on Multiple Servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "ls /path/to/directory | parallel grep pattern"'

Copying Multiple Files to Multiple Servers

parallel scp ::: file1 file2 file3 ::: server1:/path server2:/path server3:/path

Copying Multiple Directories to Multiple Servers

parallel scp -r ::: dir1 dir2 dir3 ::: server1:/path server2:/path server3:/path

Running a Command on Each Line of a File on Multiple Servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "cat /path/to/file | parallel command"'

Backing Up a Directory to Multiple Servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "tar -czf /backup_dir/backup.tar.gz /path/to/directory"'

Updating a Package on Multiple Servers

echo -e "server1\nserver2\nserver3" | parallel -j0 --tag ssh {} sudo apt-get update