Over the weekend, a client implemented security hardening on their production LPARs. They used AIX 6.1 Security Expert. Apart from some users who had been locked out due to weak passwords, testing went well ... until about 9am Monday, when some users reported they couldn't log in.
Here were the symptoms:
- users couldn't log in after about 9am when things started to get busy
- existing users couldn't establish additional login sessions
- sshd was running, but ssh (port 22) didn't even get a login prompt
- telnetd was running (app requirement), but it also didn't get a login prompt
- ftp (port 21) worked
- console access worked
- there were 257 users logged in (including a console user)
only the sharpest mathematical minds would pick up:
257 is suspiciously close to 256!
A mathematical experiment
Gather together 11.8 friends - and yourself - and all take off gloves and shoes and socks to confirm the abovementioned amazing mathematical fact. You may need to find another 12.8 friends - if you have them - to do the comparison.
Well, one console session was accounted for, and the rest came down to a limit on the number of network connections allowed. Enter the protagonist of this drama, the AIX 6.1 tcptr command. That is the command which allows you to regulate the number of connections allowed for a range of ports.
The telnet port 23 and ssh port 22 were in the same range for tcptr. AND their limit was set to 256, as shown by the tcptr -show command. On top of that, ftp was permitted because it was on port 21 - a different range, with only one connection in use.
The tcptr command had been called by aixpert.
TCP Traffic Regulation Policies:
StartPort=1 EndPort=12 MaxPool=256 Div=1 Used=0
StartPort=13 EndPort=13 MaxPool=256 Div=1 Used=0
StartPort=14 EndPort=20 MaxPool=256 Div=1 Used=0
StartPort=21 EndPort=21 MaxPool=256 Div=1 Used=1 <= ftp
StartPort=22 EndPort=25 MaxPool=256 Div=1 Used=256 <= includes ssh (port 22) and telnet (port 23)
StartPort=26 EndPort=36 MaxPool=256 Div=1 Used=0
StartPort=37 EndPort=37 MaxPool=256 Div=1 Used=0
StartPort=38 EndPort=110 MaxPool=512 Div=1 Used=0
StartPort=111 EndPort=111 MaxPool=256 Div=1 Used=0
StartPort=112 EndPort=9089 MaxPool=512 Div=1 Used=3
StartPort=9090 EndPort=9090 MaxPool=10 Div=3 Used=0
StartPort=9091 EndPort=65535 MaxPool=512 Div=1 Used=0
Fixing the problem was easy, once I understood the syntax of the tcptr command:
Once I increased the maximum connections for the port range StartPort=22 to EndPort=25, users were able to log in immediately. Which was just as well, because the Help Desk was then able to log new calls, including an environmental issue to do with a socks audit and bare feet in the office.
Once you've put your gloves on again, have a read of the man page for the tcptr command, and a developerWorks article on IBM AIX TCP Traffic Regulation.
Check the wiki on aixpert to find out more about SOX-COBIT compliance.