Technical Blog Post
Abstract
tacmd sudden login failure.
Body
tacmd sudden login failure
This was an issue seen with one customer, suddenly the tacmd login failed with the message
tacmd login fails - Error message "KUIC00003E"
Investigation found that this would suddenly start to happen in the running environment when up to that point in time tacmd commands had been running fine.
Tracing was set on the tacmd script:
/opt/IBM/ITM/bin/tacmd
KBB_RAS1=ERROR(UNIT:KDY ALL)(UNIT:KSH ALL)(UNIT:KGL ALL)
and on the TEMS
KBB_RAS1: ERROR (UNIT:KSH ALL) (UNIT:KUI ALL) (UNIT:KDY ALL)
At the next failure a tacmd login was done and the pdcollect was collected .
This showed the tacmd error of:
kuiras log
(Wed Mar 30 16:20:00 2016.1340-1:kuitacmdmain.cpp,5144,"getPortFromCMS")
Response ==><<HTML><HEAD><META HTTP-EQUIV='Refresh' CONTENT='2;
URL=login.htm'></META><TITLE>Error</TITLE></HEAD><BODY><H1>Error
401</H1> <P>Invalid user ID or password.</P></BODY></HTML>>
(Wed Mar 30 16:20:00 2016.1341-1:kuitacmdmain.cpp,5195,"getPortFromCMS")
Unable to get Service Point for retrieval of TEMS from Response
(Wed Mar 30 16:20:00 2016.1342-1:kuitacmdmain.cpp,5203,"getPortFromCMS")
Could not get alternate port number from TEMS
and TEMs logs of:
:kdhscsv.c,137,"login_attempts_exceeded") failedLogins 5
maxFailedLogins 5 currentTime 1454944800 lastLoginFailure 1454944252
(Wed Mar 30 16:20:00 2016.033A-1A:kshstrt.cpp,127,"default_service")
Login attempts exceeded
This indicated that the tacmd command had hit the number of login attempts allowed by one ip and a lock out for the id had been set.
Investigation found that tacmd commands were being run every minute from scripts.
This was found to be too high a level, once this was lowered to every 10 minutes that problems were no longer seen.
It should be noted that running tacmd commands use a SOAP login and therefore are checked for how many over a period is done and if there are more than one script running this issue can be hit.
Also the requirement to run tacmd commands on the system should be reviewed as these, like any other requests do take resources and so running many requests can cause unforeseen issues due to the load on the environment.
UID
ibm11084107