85
Recent releases of AIX installation media (for 7.1 and 6.1) now contain the OpenSSH base installation filesets. This is very handy; we no longer need to download or locate the software from other sources.
One thing to consider is what this means for future AIX migrations.
If you are migrating a system (that already has a version of SSH installed) to AIX 7.1 then you may notice that the first time you attempt to connect to the server (after the 7.1 migration) the following ssh message appears:
root@nim1 : / # ssh aixlpar1
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
59:68:05:71:60:b5:d1:96:87:df:f6:9c:ca:9a:14:3e.
Please contact your system administrator.
Add correct host key in /.ssh/known_hosts to get rid of this message.
Offending RSA key in /.ssh/known_hosts:17
RSA host key for aixlpar1 has changed and you have requested strict checking.
Host key verification failed.
In the output above I’m attempting to SSH from another system to the newly migrated AIX 7.1 LPAR. This is essentially informing us that the SSH host keys on the AIX 7.1 server don’t match the host key stored in the local systems /.ssh/known_hosts file. Something has changed.
Now of course I could simply accept this change and update my known_hosts files, like so:
root@nim1 : / # ssh-keygen -R aixlpar1
/.ssh/known_hosts updated.
Original contents retained as /.ssh/known_hosts.old
With known_hosts updated, I’m able to SSH to the AIX 7.1 system successfully.
cgibson@nim1 : /home/cgibson $ ssh aixlpar1 date
Mon Aug 20 19:44:20 EET 2012
But that’s just for my SSH known_hosts file only. What about all the users that connect to this system via SSH/SFTP/SCP? Do I really expect all of them to update their known_hosts file with the new host key information?
This could create problems for automated tasks, like file transfers. If these transfers stop working then their could be “hell to pay”. So the question I’m often asked is what can I do to prevent this from happening in the first place? Luckily there is a way.
In this example, we are using nimadm to migrate from AIX 5.3 to 7.1. The AIX 7.1 lpp_source resource was created using the AIX 7.1 installation media DVDs. All filesets were copied from the DVDs, verbatim, to the new 7.1 lpp_source resource on the NIM master.
First we verify that the openssh* filesets are in fact in the AIX 7.1 lpp_source on the NIM master.
root@nim1 : / # nim -o showres lpp_sourceaix710101 | grep -i ssh
openssh.base.client 5.4.0.6100 I N usr,root
openssh.base.client 5.8.0.6101 I N usr,root
openssh.base.server 5.4.0.6100 I N usr,root
openssh.base.server 5.8.0.6101 I N usr,root
openssh.man.en_US 5.4.0.6100 I N usr
openssh.man.en_US 5.8.0.6101 I N usr
openssh.msg.EN_US 5.8.0.6101 I N usr
openssh.msg.en_US 5.4.0.6100 I N usr
openssh.msg.en_US 5.8.0.6101 I N usr
On the NIM client (running AIX 5.3), we verify there is an older version of SSH already installed. The migration will remove these filesets (and the associated /etc/ssh_host_* files). The newer version of SSH will be installed and new ssh_host_key* files will be generated (hence the problem with the remote SSH clients known_hosts files no longer holding the correct host keys).
aixlpar1 : / # oslevel -s
5300-12-04-1119
aixlpar1 : / # lslpp -l openssh*
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
openssh.base.client 4.3.0.5301 COMMITTED Open Secure Shell Commands
openssh.base.server 4.3.0.5301 COMMITTED Open Secure Shell Server
openssh.license 4.3.0.5301 COMMITTED Open Secure Shell License
openssh.man.en_US 4.3.0.5301 COMMITTED Open Secure Shell
Documentation - U.S. English
The following SSH files are regenerated during the new fileset install:
aixlpar1 : /home/cgibson/ssh_orig # ls -ltr /etc/ssh/ssh_host*
-rw-r--r-- 1 root system 210 Feb 03 2006 /etc/ssh/ssh_host_rsa_key.pub
-rw------- 1 root system 887 Feb 03 2006 /etc/ssh/ssh_host_rsa_key
-rw-r--r-- 1 root system 319 Feb 03 2006 /etc/ssh/ssh_host_key.pub
-rw------- 1 root system 515 Feb 03 2006 /etc/ssh/ssh_host_key
-rw-r--r-- 1 root system 590 Feb 03 2006 /etc/ssh/ssh_host_dsa_key.pub
-rw------- 1 root system 668 Feb 03 2006 /etc/ssh/ssh_host_dsa_key
You can confirm the filesets are being installed during the nimadm operation by reviewing your clients nimadm log file in /var/adm/ras/alt_mig:
root@nim1 : /var/adm/ras/alt_mig # grep –i openssh aixlpar1_alt_mig.log
openssh.base.client 5.8.0.6101 USR APPLY SUCCESS
openssh.base.server 5.8.0.6101 USR APPLY SUCCESS
openssh.base.client 5.8.0.6101 ROOT APPLY SUCCESS
openssh.base.server 5.8.0.6101 ROOT APPLY SUCCESS
openssh.man.en_US 5.8.0.6101 USR APPLY SUCCESS
openssh.msg.en_US 5.8.0.6101 USR APPLY SUCCESS
aixlpar1 : / # ls -ltr /etc/ssh
total 352
-rw-r--r-- 1 root system 210 Feb 03 2006 ssh_host_rsa_key.pub
-rw------- 1 root system 887 Feb 03 2006 ssh_host_rsa_key
-rw-r--r-- 1 root system 319 Feb 03 2006 ssh_host_key.pub
-rw------- 1 root system 515 Feb 03 2006 ssh_host_key
-rw-r--r-- 1 root system 590 Feb 03 2006 ssh_host_dsa_key.pub
-rw------- 1 root system 668 Feb 03 2006 ssh_host_dsa_key
-rw-r--r-- 1 root system 2341 Apr 26 2007 ssh_prng_cmds
-rw------- 1 root system 132839 Apr 26 2007 moduli
-rw-r--r-- 1 root system 1288 May 01 2007 ssh_config
-rw-r--r-- 1 root system 1155 May 04 2007 sshd_banner
-rw-r--r-- 1 root system 2867 Oct 29 2008 sshd_config
-rw-r----- 1 root system 7 Aug 20 16:10 sshd.pid
To work around this issue, we can move the openssh* filesets out of the lpp_source directory (installp/ppc), like so:
root@nim1 : / # lsnim -l lpp_sourceaix710101
lpp_sourceaix710101:
class = resources
type = lpp_source
arch = power
Rstate = ready for use
prev_state = ready for use
location = /export/lpp_source/lpp_sourceaix710101
simages = yes
alloc_count = 0
server = master
root@nim1 : / # cd /export/lpp_source/lpp_sourceaix710101
root@nim1 : /export/lpp_source/lpp_sourceaix710101 # find . | grep -i openssh
./installp/ppc/openssh.base
./installp/ppc/openssh.man.en_US
./installp/ppc/openssh.msg.en_US
./installp/ppc/openssh.msg.en_US.5.8.0.6101.I
./installp/ppc/openssh.msg.EN_US.5.8.0.6101.I
./installp/ppc/openssh.man.en_US.5.8.0.6101.I
./installp/ppc/openssh.base.5.8.0.6101.I
root@nim1 : /export/lpp_source/lpp_sourceaix710101 # cd ./installp/ppc/
root@nim1 : /export/lpp_source/lpp_sourceaix710101/installp/ppc # ls -ltr openssh*
-rw-r----- 1 root system 14336 Nov 16 2011 openssh.msg.en_US
-rw-r----- 1 root system 138240 Nov 16 2011 openssh.man.en_US
-rw-r----- 1 root system 6024192 Nov 16 2011 openssh.base
-rw-r----- 1 root system 17408 Nov 16 2011 openssh.msg.en_US.5.8.0.6101.I
-rw-r----- 1 root system 17408 Nov 16 2011 openssh.msg.EN_US.5.8.0.6101.I
-rw-r----- 1 root system 153600 Nov 16 2011 openssh.man.en_US.5.8.0.6101.I
-rw-r----- 1 root system 6298624 Nov 16 2011 openssh.base.5.8.0.6101.I
I’ve chosen to move the filesets to a different directory (notice I’ve created a .toc file in the new location). I also need to copy (not move) the openssl* filesets to the new location as well.
root@nim1 : /export/lpp_source/lpp_sourceaix710101/installp/ppc # mkdir /export/lpp_source/lpp_sourceaix710101/ssh
root@nim1 : /export/lpp_source/lpp_sourceaix710101/installp/ppc # mv openssh* /export/lpp_source/lpp_sourceaix710101/ssh
root@nim1 : /export/lpp_source/lpp_sourceaix710101/installp/ppc # cp -p openssl* /export/lpp_source/lpp_sourceaix710101/ssh
root@nim1 : / # cd /export/lpp_source/lpp_sourceaix710101/ssh
root@nim1 : /export/lpp_source/lpp_sourceaix710101/ssh # inutoc .
root@nim1 : /export/lpp_source/lpp_sourceaix710101/ssh # ls -ltr
total 58648
-rw-r----- 1 root system 6298624 Nov 16 2011 openssl.base
-rw-r----- 1 root system 14336 Nov 16 2011 openssh.msg.en_US
-rw-r----- 1 root system 138240 Nov 16 2011 openssh.man.en_US
-rw-r----- 1 root system 6024192 Nov 16 2011 openssh.base
-rw-r----- 1 root system 2350080 Nov 16 2011 openssl.man.en_US
-rw-r----- 1 root system 28672 Nov 16 2011 openssl.license
-rw-r----- 1 root system 2350080 Nov 16 2011 openssl.man.en_US.0.9.8.1800.I
-rw-r----- 1 root system 3072 Nov 16 2011 openssl.license.0.9.8.1800.I
-rw-r----- 1 root system 6307840 Nov 16 2011 openssl.base.0.9.8.1800.I
-rw-r----- 1 root system 17408 Nov 16 2011 openssh.msg.en_US.5.8.0.6101.I
-rw-r----- 1 root system 17408 Nov 16 2011 openssh.msg.EN_US.5.8.0.6101.I
-rw-r----- 1 root system 153600 Nov 16 2011 openssh.man.en_US.5.8.0.6101.I
-rw-r----- 1 root system 6298624 Nov 16 2011 openssh.base.5.8.0.6101.I
-rw-r--r-- 1 root system 4966 Aug 20 20:41 .toc
Now that openssh is no longer in the lppsource, I check the AIX 7.1 lpp_source and update its TOC:
root@nim1 : / # nim -o check lpp_sourceaix710101
The openssh filesets are no longer listed as part of the lppsource:
root@nim1 : / # nim -o showres lpp_sourceaix710101 | grep -i ssh
root@nim1 : / #
At this point we perform a standard nimadm operation to migrate from AIX 5.3 to 7.1. Openssh is not installed during the migration:
root@nim1: /var/adm/ras/alt_mig # grep -i openssh aixlpar1_alt_mig.log
root@nim1: /var/adm/ras/alt_mig #
After the migration, ssh connections to the server function just as they did before the migration.
cgibson@nim1 : /home/cgibson $ ssh aixlpar1 date
Mon Aug 20 20:36:31 EETDT 2012
Now we can update the openssh filesets to the newer level. This will install the newer filesets but preserve the existing SSH host keys.
aixlpar1 : / # mount nim1:/export/lpp_source/lpp_sourceaix710101/ssh /mnt
aixlpar1 : / # installp -Y -a -d /mnt ALL
...etc...
Installation Summary
--------------------
Name Level Part Event Result
-------------------------------------------------------------------------------
openssl.man.en_US 0.9.8.1800 USR APPLY SUCCESS
openssl.license 0.9.8.1800 USR APPLY SUCCESS
openssl.base 0.9.8.1800 USR APPLY SUCCESS
openssl.base 0.9.8.1800 ROOT APPLY SUCCESS
openssh.base.client 5.8.0.6101 USR APPLY SUCCESS
openssh.base.server 5.8.0.6101 USR APPLY SUCCESS
openssh.base.client 5.8.0.6101 ROOT APPLY SUCCESS
openssh.base.server 5.8.0.6101 ROOT APPLY SUCCESS
openssh.msg.en_US 5.8.0.6101 USR APPLY SUCCESS
openssh.msg.EN_US 5.8.0.6101 USR APPLY SUCCESS
openssh.man.en_US 5.8.0.6101 USR APPLY SUCCESS
Rather than update these filesets manually after the migration, you can include this step as a post migration task with nimadm.
An alternative way to work around this problem (after the fact) would be to restore the original ssh_host_key* files from a backup. For example, I copied the original ssh_host_key* files to my home directory before starting the AIX migration.
aixlpar1 : / # cd /etc
aixlpar1 : /etc # cp -pr ssh /home/cgibson/ssh_orig/
In the output below, I discover that my ssh_host_key* files have all been recreated during the migration.
aixlpar1 : /etc/ssh # ls -ltr
total 352
-rw-r--r-- 1 root system 1288 May 01 2007 ssh_config
-rw-r--r-- 1 root system 1155 May 04 2007 sshd_banner
-rw-r--r-- 1 root system 2867 Oct 29 2008 sshd_config
-rw-r----- 1 root system 7 Aug 20 21:00 sshd.pid
-rw-r--r-- 1 root system 2341 Aug 20 21:19 ssh_prng_cmds
-rw------- 1 root system 132839 Aug 20 21:19 moduli
-rw-r----- 1 root system 382 Aug 20 21:45 ssh_host_rsa_key.pub
-rw------- 1 root system 1679 Aug 20 21:45 ssh_host_rsa_key
-rw-r----- 1 root system 630 Aug 20 21:45 ssh_host_key.pub
-rw------- 1 root system 965 Aug 20 21:45 ssh_host_key
-rw-r----- 1 root system 590 Aug 20 21:45 ssh_host_dsa_key.pub
-rw------- 1 root system 668 Aug 20 21:45 ssh_host_dsa_key
I copy the original files back to the /etc/ssh directory. The sshd subsystem is also restarted to pick up the updated ssh_host* files.
aixlpar1 : /etc/ssh # cp -p /home/cgibson/ssh_orig/ssh_host_* .
aixlpar1 : /etc/ssh # ls -ltr
total 352
-rw-r--r-- 1 root system 210 Feb 03 2006 ssh_host_rsa_key.pub
-rw------- 1 root system 887 Feb 03 2006 ssh_host_rsa_key
-rw-r--r-- 1 root system 319 Feb 03 2006 ssh_host_key.pub
-rw------- 1 root system 515 Feb 03 2006 ssh_host_key
-rw-r--r-- 1 root system 590 Feb 03 2006 ssh_host_dsa_key.pub
-rw------- 1 root system 668 Feb 03 2006 ssh_host_dsa_key
-rw-r--r-- 1 root system 1288 May 01 2007 ssh_config
-rw-r--r-- 1 root system 1155 May 04 2007 sshd_banner
-rw-r--r-- 1 root system 2867 Oct 29 2008 sshd_config
-rw-r----- 1 root system 7 Aug 20 21:00 sshd.pid
-rw-r--r-- 1 root system 2341 Aug 20 21:19 ssh_prng_cmds
-rw------- 1 root system 132839 Aug 20 21:19 moduli
aixlpar1 : /etc/ssh # stopsrc -s sshd
0513-044 The sshd Subsystem was requested to stop.
aixlpar1 : /etc/ssh # startsrc -s sshd
0513-059 The sshd Subsystem has been started. Subsystem PID is 3997822.
