5 Steps to updating OpenSSH on a VIO server
I was doing some health checks when I noticed that some of our VIO (2.1.3) servers was running a version of OpenSSH that has a known security vulnerability. You will hopefully be aware that OpenSSH is provided by IBM as part of the AIX since v6.1 which means that the latest TL and VIO updates will include fixes for SSH and any related security issues.
I will be quick to acknowledge that the simplest approach might be to ensure that regular VIOS software maintenance is instated and to regularly perform VIO server updates of the entire system. But in many production environments it is not possible to do this without extensive planning and delay, so the alternative I would like to discuss is an update of just OpenSSH itself.
What follows here is an outline of a reasonably straightforward process for updating just the IBM OpenSSH software without having to update any other software. Please however note that the file set names and versions used do not apply to the portable version of OpenSSH.
1. Determine if you are running a vulnerable or weak level of OpenSSH (or the dependent OpenSSL file set) that requires an update.
There are various sources to identify SSH vulnerabilities, and I based my information on to http://www.openssh.org/security.html. This stipulates that Portable OpenSSH prior to version 5.8p2 is vulnerable to a local
host key theft attack described in
To compare this with the installed level of OpenSSH, run:
root:# lslpp -lc openssh.base.server
#Fileset:Level:PTF Id:State:Type:Description:EFIX Locked
/usr/lib/objrepos:openssh.base.server:18.104.22.16801::COMMITTED:I:Open Secure Shell Server:
/etc/objrepos:openssh.base.server:22.214.171.12401::COMMITTED:I:Open Secure Shell Server:
Oh dear. This system is making use of OpenSSH 5.2 and would be at risk against a local/internal network attack, however unlikely that may be.
2. Decide on a new OpenSSH file set level.
The typical approach might be to navigate to the OpenSSH on AIX
SourceForge page but I elected not to do so since I am more trusting of the actual software included in the AIX technology level and service pack updates.
An (easy/safe) approach is to navigate to the Fileset Information for openssh.base.server
page and select the highest level of OpenSSH for the current release of AIX that is active the VIO server. In other words, I would deem openssh.base.server.126.96.36.19900 (in AIX 6100-08) as the best update choice, since it is the highest level of SSH available for AIX 6.1 that runs on a VIO 2.1.3 server. I would however never consider a file-set for AIX 7.1 as a candidate on a AIX 6.1 system, even though it may technically happen to be the same.
At this point, you would have to download the service pack or technology level if you have not already done so. Since I have various packages available at my disposal on my NIM servers, I was able to simply navigate to the appropriate directory before continuing.
3. Commit all existing file sets on the VIO server.
It is essential to ensure that all software changes are revertible - this is the golden rule of system administration. I ensured that I have a valid backup of the VIO server before proceeding to commit all existing file sets, as the safest approach to the update would be to only APPLY the new SSH file sets.
To do so, I ran:
Commit Applied Software Updates (Remove Saved Files)
and this committed about 260 file-sets which remained as a backout point from a previous VIO server update in short amount of time.
4. Copy the required OpenSSH and OpenSSL file sets to a temporary directory.
I found that it is not possible to simply select and update just the SSH specific file sets from the AIX TL repository, as a preview operation (be safe) indicated that automatic requisite installation would require an update of other file sets like bos.rte.install. While obvious, I will rather also mention that under no circumstances should one attempt to update a VIO server directly using any AIX technology level software - the only valid upgrade source for a VIO server is the VIO server specific service packs.
Now, to identify the installed openssh and openssl packages that must be updated:
vio1:/# lslpp -l|egrep -e "openss[hl]"
At which point I was able to identify the following file-sets to update (language specific file sets may vary):
openssl.license (needed for licensing purposes)
openssl.base (includes server and client software)
To identify names of the actual BFF files that contain the updates, I simply had to run a grep command for each of the above in the .toc, for example:
vio1:/mnt/aix6/6100-07-05-1228# grep "openssh.base.server" .toc
At this point, I was able to identify and copy the file sets to a temporary location:
/mnt/aix6/6100-07-05-1228# mkdir /tmp/openssh && cp -p U846944.bff /tmp/openssh
and after repeating the copy for each of the above file sets I could execute
inutoc . from within the temporary directory.
5. Update OpenSSH.
Updating the file sets is straightforward via
smitty update_all, except for a few additional SSH related precautions.
The thing is, in the worst case scenario it is conceivable that the SSH update fails and that the SSH daemon may abort for an inexplicable reason. In such a case only existing SSH sessions will work, new connections can not be made unless the sshd daemon is active. So it is always a good idea to establish a separate SSH connection as a fall back (and to make sure that the TMOUT setting does not automatically log you out). This session will not be terminated if the sshd itself aborted during the update process.
Secondly, the safest approach would also be to back up the SSH keys, authorized_keys, known_hosts and all other SSH configuration files in ~root/.ssh in case anything gets overwritten, so run:
tar cvf /tmp/root-ssh.tar ~root/.ssh
Okay, enough fooling around. In order to perform the actual update - I first perform a preview update (as per any software operation I do), and then perform the actual installation:
smitty update_all (select the current directory as the input device)Now, all that is required is a restart of the SSH daemon. It is managed by the SRC, and can be restarted by the following commands:
PREVIEW only? YES OR NO
COMMIT software updates? NO
AUTOMATICALLY install requisite software? YES
ACCEPT new license agreements? YES
stopsrc -s sshd; sleep 3; startsrc -s sshd;
and then testing a new SSH connection to the system. The process associated with this new SSH connection should have a later start time than the actual sshd parent process that we just restarted. Do not forget to restart the SSH daemon, or else you might get a nasty surprise when you reboot the system (if something went wrong).
In conclusion, this is a very straightforward update process that I approached rather cautiously to be safe. It is actually best to test an update like this on a non-critical environment, but it is not very challenging and it may be useful in case you require the update for compliance or other reasons of irritation.
Till next time!