IV61449: IMS SERVER IS NOT RESPONDING HUNG THREADS AND FAILED TO GET A TICKET FOR GETPUBLICATTRIBUTESBYBAG NOTED IN THE LOGS

APAR status

• Closed as program error.

Error description

• We are currently having a production outage of IBM Security
  Access Manager for Enterprise Single Sign-On 8.2.  In the
  Systemout.log we have noticed two reoccurring errors:
  
  faultString: java.io.IOException: Async IO operation failed
  (1), reason: RC: 55  The specified network resource or device
  is no longer available.
  
  ThreadMonitor W   WSVR0605W: Thread "TCPChannel.DCS : 1"
  (00000013) has been active for 744306 milliseconds and may be
  hung.  There is/are 47 thread(s) in total in the server that
  may be hung.
  
  We rebooted the 2 production servers to see if that would clear
  up the issues, and it only did temporarily, then the number of
  threads hung grew again and the application is now
  unresponsive.  Attaching logs
  
  Customer has upgraded from 8.1 to 8.2 about 4 weeks ago.
  There were recently profile changes made and uploaded to the
  IMS Server.
  
  We are seeing a couple different errors in the logs
  1. Deadlocks in SQL Server. These show as exceptions and hung
  threads reading from SQL Server.
  
  2. Web services socket read errors:
  java.io.IOException: Async IO operation failed (1), reason: RC:
  55  The specified network resource or device is no longer
  available.
  
  After learning that the issue first came up after a profile or
  profiles had been uploaded this morning we checked the config
  files for the download throttling server and there was no entry
  for this.
  Following a profile upload and discovering the client did not
  have throttling entris in the ims.xml file the following
  entries were made.
  
  <encentuate.ims.service.downloadService.maxThread>
  <value xml:lang="en">10</value>
  
  </encentuate.ims.service.downloadService.maxThread>
  >>> 10 is the recommended value
  
  <encentuate.ims.service.downloadService.timeout.seconds>
  <value xml:lang="en">30</value>
  
  Cluster was stopped and restarted, we monitored the system for
  10 minutes and IMS response appeared to be normal.
  
  IMS Server had been responding normal following this until 4/22
  when they noticed IMS Server was not responding again.
  At 6:43 we see [4/22/14 6:43:05:057 EDT] 00000066 DownloadServi
  W   Failed to get a ticket for getPublicAttributesByBag
  6:50 am on 4.22 lasting until 7:02 where 44 threads may have
  been hung.  This cleared up at 7:02 and the IMS Server had
  resumed normal operations.
  

Local fix

• Uploading existing access profiles (one profile at a time; for
  each profile):
  a) make a copy of the final copy of the access profile (.eas
  file)
  b) open the copy of the access profile from step (a) in Access
  Studio
  c) go to [View] > [Authentication Services], right-click on
  each of the authentication services(which (exists) had
  previously already been uploaded to IMS) and delete them
  d) go to [View] > [AccessProfiles], right click on the profile
  and select "Upload to IMS"
  e) remove the copy of the access profile from step (a) (.eas
  file)
  

Problem summary

• Enhancement made to Access Studio.
  

Problem conclusion

• iFix provided to customer.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TAM ESSO IM SVR

• Fixed component ID

  5724V6700

Applicable component levels

• R820 PSY

     UP