SSH delay with security.pkcs11 installed.
cggibbo 270000TMUJ Comments (6) Visits (23579)
After updating my AIX 7.1 TL3 system to service pack 4, I noticed that each time I started a new ssh session with this system, there was a noticeable delay before the login prompt was displayed. I initially thought there was a network or host name resolution (DNS) problem, but after thoroughly checking related files, such as /etc/hosts, /etc/resolv.conf and /etc/netsvc.conf, I started looking for a problem elsewhere.
I used truss to assist me in my investigation. I found that the each time an ssh client connected to the sshd daemon, sshd would attempt to access a device named /dev/pkcs11. Each time this happened (once per login) there was a significant delay/pause before the ssh session continued to the login prompt. I also noticed that prior to applying SP4, this delay wasn’t present.
I ran truss with the following options (the -d flag provided me with a timestamp for each line of output, and helped my detect the delay!). Immediately after the pkcs11 device was opened, there was a several second delay before the process continued.
# truss -d ssh lpar9 date > cg.out 2>&1
4.1939: kioctl(3, 2, 0x2FF21848, 0x00000000) = 0
# time ssh lpar9 date
Wed Jan 14 15:15:04 2015
# oslevel -s
# lslpp -l security.pkcs11
I decided to un-install the security.pkcs11 fileset. This solved the issue and my ssh sessions started quickly with the login prompt appearing instantly again.
# installp -u security.pkcs11 -g
# time ssh lpar9 date
Wed Jan 14 15:13:56 2015
I’m still not sure what caused this problem. Prior to SP4, I did not encounter this issue with the security.pkcs11 fileset installed, so I can only assume that there may be some issue with this fileset at the 18.104.22.168 level. Here’s the truss output from a system running a lower level of security.pkcs11 (no delay).
0.1320: kioctl(3, 2, 0x2FF21848, 0x00000000) = 0
I also found some advice that suggested placing ‘UsePKCS no’ in the /etc
It was safe for me to remove this fileset as I was not using it for any purpose. Typically, this fileset is required when using special crypto cards in POWER servers.
IBM 4758 Model 2 Cryptographic Coprocessor
Public Key Cryptography Standards #11
Hopefully this will help others that may encounter this problem on their AIX systems.