NIM Master, NIMSH and SSL on AIX 7.1 TL4 SP3
cggibbo 270000TMUJ Visits (12071)
Whilst working with one of my AIX customers recently I discovered a problem with NIMSH and SSL. The customer had updated their NIM master from AIX 7.1 TL4 SP1 to AIX 7.1 TL4 SP3. After the SP update, any attempt to connect to a NIM client (over NIMSH+SSL), from the NIM master, would simply hang. For example, we tried to list the filesets on the NIM client with this command, which never returned any output.
[root@750lpar4]/ # nim -o lslpp 750lpar9
Thu Jan 12 14:31:49 2017 Loading certificates..
Thu Jan 12 14:31:49 2017 Loading private key file..
Thu Jan 12 14:31:49 2017 create BIO
NIM master: 750lpar4
NIM client: 750lpar9
[root@750lpar4]/ # lsnim -l 750lpar9
class = machines
type = standalone
connect = nimsh (secure)
platform = chrp
netboot_kernel = 64
if1 = 10_1_50 750lpar9 0
cable_type1 = N/A
Cstate = ready for a NIM operation
prev_state = not running
Mstate = currently running
cpuid = 00F603CD4C00
Cstate_result = success
The root cause of the problem become apparent when we ran truss against the nim –o command.
[root@750lpar4]/ # truss -adef -o truss.lsnim.out -w all nim -o lslpp 750lpar9
[root@750lpar4]/ # cat truss.lsnim.out
13959372: C o u l d n o t l o a d m o d u l e / u s r / l i b / l
13959372: i b s s l . s o .\n S y s t e m e r r o r : N o s u c h
13959372: f i l e o r d i r e c t o r y
19267612: C o u l d n o t l o a d m o d u l e / u s r / l i b / l
19267612: i b c r y p t o . s o .\n S y s t e m e r r o r : N o s u
19267612: c h f i l e o r d i r e c t o r y
The required shared library object files were missing on the NIM master.
libssl.so not found
libcrypto.so not found
We fixed this issue by extracting the missing files from the (existing) /usr/lib/libssl.a and /usr
x - /usr/lib/libssl.so
x - /usr
-rwxr-xr-x 1 root system 724913 Jan 18 09:08 libssl.so
-rwxr-xr-x 1 root system 3031337 Jan 18 09:08 libcrypto.so
After that, the nim –o commands started working again.
So, the question is why did this happen? Well, in the past, the libssl.so.0.9.8 shared object was extracted by NIM, but more recent updates by the OpenSSL version has forced IBM to move to libssl.so. Usually, the extracted shared library object is added (if not currently present) when nimconfig -c is run. But given that this is an existing NIM master, we did not want to run this again (as we would lose all of the current SSL key access). So extracting the objects is preferred. The problem is due to the fact that the libssl.so and libcrypto.so files are not populated when the AIX 7100-04-03 update is applied. This is a bug and will be officially addressed, soon, under APAR IV93152 NIM push operation to client hang on nimsh over SSL.
I believe this issue may also occur when you migrate your NIM master from AIX 7.1 to 7.2 (with nimadm for example). But I need to do more testing to reproduce and confirm the issue.
Here’s one good reason to setup NIMSH over SSL.
NIMSH, SSL and LPM
The following link is a great reference guide for configuring NIMSH over SSL.
NIMSH over SSL