How to re-generate ssh key pair to enable passwordless login to compute nodes

Resolving The Problem

Platform HPC has configured the passwordless login for all users from the installer node to all the compute nodes automatically. In some cases, the files required for passwordless login are lost, this article describes how to troubleshoot and fix this issue manually.

The first time you log into a user account on the installer, you should see something like this :

---------------------------------------------------------- --------------------------
It doesn't appear that you have set up your ssh key.
This process will make the files:
/home/test1/.ssh/id_rsa.pub
/home/test1/.ssh/id_rsa
/home/test1/.ssh/authorized_keys
Generating public/private rsa key pair.
Enter file in which to save the key (/home/test1/.ssh/id_rsa):
------------------------------------------------------------------------------ ------

You should press Enter 3 times :

---------------------------------------------------------- ---------------
Created directory '/home/test1/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/test1/.ssh/id_rsa.
Your public key has been saved in /home/test1/.ssh/id_rsa.pub.
The key fingerprint is:
26:51:cb:ae:dc:08:7c:c5:2e:b4:37:56:50:f2:4b:2f test1@headnode.com
-------------------------------------------------------------------------

Then, you can run the pdsh -a command. The first time, the compute nodes will be added to the known_hosts file (in your ~/.ssh directory)

----------------------------------------------------- -------------------
[test1@headnode ~]$ pdsh -a uptime
compute-0-0:
Warning: Permanently added 'compute-0-0' (RSA) to the list of known hosts.
18:10:23 up 4 days, 1:02, 1 user, load average: 0.00, 0.00, 0.00
compute-0-17:
Warning: Permanently added 'compute-0-17' (RSA) to the list of known hosts.
17:10:24 up 3 days, 7:21, 0 users, load average: 0.03, 0.03, 0.00
------------------------------------------------------------------------
NOTE:
If you choose to type a specific password the first time, you will have to type it all the time (for every ssh connection). This will disable the passwordless login, which is required for MPI job execution. So you need to use an empty passphrase in this case. This empty passphrase will be only used inside the cluster.

 

If for some reasons, you cannot login passwordless to a compute node, please check the following files exist under $HOME/.ssh:

 

-rw-r--r-- 1 lsfadmin lsfadmin 415 Feb 14 2011 authorized_keys
-rw------- 1 lsfadmin lsfadmin 1675 Feb 14 2011 id_rsa
-rw-r--r-- 1 lsfadmin lsfadmin 415 Feb 14 2011 id_rsa.pub
-rw-r--r-- 1 lsfadmin lsfadmin 409 Mar 22 2011 known_hosts

 

You need to make sure that the contents of id_rsa.pub (public key for the user on the installer) is included in the authorized_keys (granting passwordless access).

If you want to restore the files in $HOME/.ssh directory, you can re-generate the SSH key pair and get the passwordless ssh connection again this way :
(1) move the old .ssh directory : mv .ssh/ .ssh.old
(2) create a new .ssh directory : mkdir .ssh
(3) be sure that this new directory has the correct permissions (it is not the case by default because of the umask) : chmod 600 .ssh ; chmod u+x .ssh
(4) generate a new key : ssh-keygen -t rsa

--------------------------------------- ---------------------
Generating public/private rsa key pair.
Enter file in which to save the key (/home/test1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/test1/.ssh/id_rsa.
Your public key has been saved in /home/test1/.ssh/id_rsa.pub.
The key fingerprint is:
b1:20:4f:cf:0b:b2:1d:54:75:81:f3:48:47:07:1a:24 test1@headnode.com
------------------------------------------------------------

Please note that you can also generate a DSA key. The only thing you have to pay attention is again, the empty passphrase.

5- add the new key to the authorized keys and set the correct permissions :
cd .ssh; cat id_rsa.pub > authorized_keys;chmod 600 authorized_keys

6- check that everything is ok : ssh -x compute-0-0 uptime for example

---------------------------------------------------------- --
[test1@headnode .ssh]$ ssh -x compute-0-0 uptime
18:22:44 up 4 days, 1:14, 1 user, load average: 0.00, 0.00, 0.00
------------------------------------------------------------


Please note that if the permissions are not correct, you will be prompted for password.