PSSH - 10 practical examples to use parallel SSH in Linux

PSSH is short abbreviation for Parallel Secure SHell or Parallel SSH. pssh is a program for executing ssh in parallel on a number of hosts. It provides features such as sending input to all of the processes, passing a password to ssh, saving output to files, and timing out.

The PSSH_NODENUM and PSSH_HOST environment variables are sent to the remote host. The <a href="https://www.golinuxcloud.com/openssh-authentication-methods-sshd-config/" target="_blank" rel="noopener noreferrer">PSSH_NODENUM</a> variable is assigned a unique number for each ssh connection, starting with 0 and counting up. The PSSH_HOST variable is assigned the name of the host as specified in the hosts list.

PSSH is not part of RHEL or CentOS repo, so you will need to manually download and install the rpm from EPEL repo.For the sake of this demo I have created a password less configuration between multiple nodes on my setup.

1. Pass list of hosts using a file

You can create a file with list of hosts which can be used as an input for PSSH.

# cat /tmp/host_file.txt
root@10.43.138.2:22
root@10.43.138.3:22
root@10.43.138.9:22

Now we will call PSSH for all the hosts from this file to execute date command.

# /bin/pssh -h /tmp/host_file.txt date
[1] 19:29:01 [SUCCESS] root@10.43.138.3:22
[2] 19:29:01 [SUCCESS] root@10.43.138.2:22
[3] 19:29:01 [SUCCESS] root@10.43.138.9:22

So as you see the tool was successfully able to connect all the provided hosts and the EXIT STATUS for all the hosts was SUCCESS.

2. Pass list of hosts manually

If you only have few hosts list on which you wish to execute certain commands, then you can manually pass the info

# /bin/pssh -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date
[1] 19:39:00 [SUCCESS] 10.43.138.3
[2] 19:39:00 [SUCCESS] 10.43.138.2
[3] 19:39:00 [SUCCESS] 10.43.138.9

3. Print inline output per host

You may have observed that for both the above examples, the EXIT STATUS was SUCCESS but yet the tool never printed any output on the screen. Use ‘-i’ to display standard output and standard error as each host completes.

In the below example as you see we get an output for every host once the execution completes.

# /bin/pssh -i -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date
[1] 19:39:00 [SUCCESS] 10.43.138.3
Sun Dec 16 19:39:00 IST 2018
[2] 19:39:00 [SUCCESS] 10.43.138.2
Sun Dec 16 19:39:00 IST 2018
[3] 19:39:00 [SUCCESS] 10.43.138.9
Sun Dec 16 19:39:00 IST 2018

4. Prompt for password

Since we have enabled password less communication between our hosts, the tool does not prompts for password. Using ‘-A’ the tool will prompt for a password and pass it to ssh. The password may be used for either to unlock a key or for password authentication. The password is transferred in a fairly secure manner (e.g., it will not show up in argument lists). However, be aware that a root user on your system could potentially intercept the password.

# /bin/pssh -A -i -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date
Warning: do not enter your password if anyone else has superuser
privileges or access to your account.
Password:
[1] 20:14:06 [SUCCESS] 10.43.138.3
Sun Dec 16 20:14:06 IST 2018
[2] 20:14:06 [SUCCESS] 10.43.138.2
Sun Dec 16 20:14:06 IST 2018
[3] 20:14:06 [SUCCESS] 10.43.138.9
Sun Dec 16 20:14:06 IST 2018

5. Storing the STDOUT

Using ‘-o’ or ‘--outdir’ argument you can save standard output to files in the given directory. Filenames are of the form [user@]host[:port][.num] where the user and port are only included for hosts that explicitly specify them. The number is a counter that is incremented each time for hosts that are specified more than once.

# /bin/pssh -i -o /tmp/out/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date
[1] 19:44:04 [SUCCESS] 10.43.138.3
Sun Dec 16 19:44:04 IST 2018
[2] 19:44:04 [SUCCESS] 10.43.138.2
Sun Dec 16 19:44:04 IST 2018
[3] 19:44:04 [SUCCESS] 10.43.138.9
Sun Dec 16 19:44:04 IST 2018

Next check /tmp/out to get the STDOUT result.

# ll /tmp/out/
total 12
-rw-r----- 1 root root 29 Dec 16 19:44 10.43.138.3
-rw-r----- 1 root root 29 Dec 16 19:44 10.43.138.2
-rw-r----- 1 root root 29 Dec 16 19:44 10.43.138.9

# cat /tmp/out/10.43.138.3
Sun Dec 16 19:44:04 IST 2018

6. Storing the STDERR

Using ‘-e’ or ‘--errdir’ you can save standard error to files in the given directory. Filenames are of the same form as with the ‘-o’ option.

I will repeat the same command

# /bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date
[1] 19:46:46 [SUCCESS] 10.43.138.3
Sun Dec 16 19:46:46 IST 2018
[2] 19:46:46 [SUCCESS] 10.43.138.2
Sun Dec 16 19:46:46 IST 2018
[3] 19:46:46 [SUCCESS] 10.43.138.9
Sun Dec 16 19:46:46 IST 2018

But since the EXIT STATUS for all the commands were SUCCESS, the STDERR content is NULL.

# ll /tmp/err/
total 0
-rw-r----- 1 root root 0 Dec 16 19:46 10.43.138.2
-rw-r----- 1 root root 0 Dec 16 19:46 10.43.138.3
-rw-r----- 1 root root 0 Dec 16 19:46 10.43.138.9

So for the demo let me try to get a negative output. Here I am trying to run a command 'datet' which does not exists. So this means our PSSH will throw error

# /bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root -x '-q -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes' datet
[1] 19:48:19 [FAILURE] 10.43.138.3 Exited with error code 127
Stderr: bash: datet: command not found
[2] 19:48:19 [FAILURE] 10.43.138.2 Exited with error code 127
Stderr: bash: datet: command not found
[3] 19:48:19 [FAILURE] 10.43.138.9 Exited with error code 127
Stderr: bash: datet: command not found

As expected we have received an EXIT CODE other than 0 and now if we check /tmp/err then we see the same is captured for every host.

# ls -l /tmp/err/
total 12
-rw-r----- 1 root root 31 Dec 16 19:48 10.43.138.3
-rw-r----- 1 root root 31 Dec 16 19:48 10.43.138.2
-rw-r----- 1 root root 31 Dec 16 19:48 10.43.138.9

While /tmp/out is empty since there was no STDOUT for this operation on any of the host

# ll /tmp/out/
total 0
-rw-r----- 1 root root 0 Dec 16 19:48 10.43.138.2
-rw-r----- 1 root root 0 Dec 16 19:48 10.43.138.3
-rw-r----- 1 root root 0 Dec 16 19:48 10.43.138.9

7. Use SSHD options with PSSH

Now ideally PSSH accepts only certain list of options which is supports. But this does not means you cannot use SSHD arguments with PSSH.
With ‘-x’ you can pass extra SSH command-line arguments (see the ssh(1) man page for more information about SSH arguments). This option may be specified multiple times. The arguments are processed to split on whitespace, protect text within quotes, and escape with backslashes.
To pass a single SSHD argument

# /bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root -x 'StrictHostKeyChecking=no' date
[1] 19:46:46 [SUCCESS] 10.43.138.3
Sun Dec 16 19:46:46 IST 2018
[2] 19:46:46 [SUCCESS] 10.43.138.2
Sun Dec 16 19:46:46 IST 2018
[3] 19:46:46 [SUCCESS] 10.43.138.9
Sun Dec 16 19:46:46 IST 2018

To pass multiple SSHD argument

# /bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root -x '-q -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes' date
[1] 19:53:56 [SUCCESS] 10.43.138.3
Sun Dec 16 19:53:56 IST 2018
[2] 19:53:56 [SUCCESS] 10.43.138.2
Sun Dec 16 19:53:56 IST 2018
[3] 19:53:56 [SUCCESS] 10.43.138.9
Sun Dec 16 19:53:56 IST 2018

8. Get list of hosts with SUCCESS and FAILED exit status

Now with PSSH one problem I faced was when we use this tool in scripting, we get a consolidated EXIT STATUS but we do not have host specific exits status by default. So if you were running a shell script with PSSH then you may not know out of 10 hosts which ones were successful and which one failed.

Now you can use -o and -e as an option to analyse this but this would require little scripting. I have written below piece of code in shell script which collects the lst of hosts and prints a SUMMARY with success and failed list of hosts.

function get_pssh_hosts {
     err_dir=$1
     out_dir=$2

     tmp_f_hosts=$(mktemp /tmp/tmp_f_hosts.XXX)
     tmp_s_hosts=$(mktemp /tmp/tmp_s_hosts.XXX)

     find $err_dir -type f -not -empty -print | rev | cut -d / -f 1 | rev >> $tmp_f_hosts 2>&1
     find $err_dir -type f  -empty -print | rev | cut -d / -f 1 | rev >> $tmp_s_hosts 2>&1
     if [ ! -z $out_dir ];then
        for host in `cat $tmp_s_hosts`;do
           [[ ! -f $out_dir/$host ]] && exit_with_error "Unable to get the exit status for $host"
           # If file is empty then put to failed hosts
           if [ ! -s $out_dir/$host ];then
              if ! egrep -q ^$host$ $tmp_f_hosts; then
                 /bin/echo "$host" >> $tmp_f_hosts
                 sed -i "/^$host$/d" $tmp_s_hosts
              fi
           else
              # If file is not empty then search for ERROR string, if found means the execution failed
              if ! grep -q ERROR $out_dir/$host;then
                 if ! grep -q "^$host$" $tmp_s_hosts;then
                   /bin/echo "$host" >> $tmp_s_hosts
                 fi
              # if no ERROR string found in the log then put in success and avoid duplicates
              else
                 /bin/echo "$host" >> $tmp_f_hosts
                 sed -i "/^$host$/d" $tmp_s_hosts
              fi
           fi
        done
     fi
      sed -i '/^$/d' $tmp_s_hosts
      sed -i '/^$/d' $tmp_f_hosts

      success_hosts=`cat $tmp_s_hosts | tr 'n' ' '`
      failed_hosts=`cat $tmp_f_hosts | tr 'n' ' '`

      rm -f $tmp_f_hosts
      rm -f $tmp_s_hosts
}

function print_summary {

/bin/echo ""
/bin/echo "#####"
/bin/echo "#"
/bin/echo -e "# e[01;37mSUMMARY:e[0m"
/bin/echo "#"
/bin/echo -e "# e[01;32mSuccess:e[0m $success_hosts"
/bin/echo -e "# e[01;31mFailed:e[0m $failed_hosts"
/bin/echo "#"
/bin/echo "#####"
/bin/echo ""

}

How this works?

err_dir=$(mktemp -d /tmp/errdir.XXX)
out_dir=$(mktemp -d /tmp/outdir.XXX)

/bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root `datet || echo "ERROR"`
get_pssh_hosts $err_dir $out_dir
print_summary

Now here we get output like below in STDERR

# grep ERROR /tmp/err/10.43.138.3
bash: ERROR: command not found

#####
#
# SUMMARY:
#
# Success: 
# Failed: 10.43.138.2 10.43.138.3 10.43.138.9
#
#####

So our script can easily detect and distinguish between success and failed scenario

9. Use PSCP to copy files from one server to multiple server

pscp is a program for copying files in parallel to a number of hosts. It provides features such as passing a password to scp, saving output to files, and timing out. This tool also supports almost all the options as with PSSH.

In the below example I am copying a file available on my server at ‘/tmp/temp.txt’ on the client nodes under ‘/home/deepak/’

# /bin/pscp.pssh -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root /tmp/temp.txt /home/deepak/
[1] 20:22:47 [SUCCESS] 10.43.138.3
[2] 20:22:47 [SUCCESS] 10.43.138.2
[3] 20:22:47 [SUCCESS] 10.43.138.9

10. Perform PSSH without password prompt

You can follow Perform SSH public key authentication with PSSH (without password) in Linux to get the steps to perform SSH to multiple hosts using private public key authentication. You can use this method in scripts to automate some tasks without getting any password prompt.

Summary

PSSH is a very handy tool when you want to perform any task on multiple target nodes in parallel. You can easily integrate this tool with any other wrapper script in any other programming language. I have already given one example of shell script where I am customizing the output and storing the list of success and failed hosts.

Further Reading

man page for PSSH