IC94804: V6.3 SERVER ON AIX POWER 7 SYSTEMS CAN EXPERIENCE CLIENT PERFORM ANCE DEGRADATION.

IBM Tivoli Storage Manager V6.3 Fix Pack 5 (6.3.5.000) Server Downloads
IBM Tivoli Storage Manager V7.1 Fix Pack 1 (7.1.1.000) Server Downloads

APAR status

• Closed as program error.

Error description

• Server users on AIX Power 7 (P7) systems can experience client
  backup and restore performance degradation.  Systems with a
  high rate of client connection activity (for example, thousands
  of sessions from HSM, TDP, CM, and CMOD workloads) are more
  likely to to be affected by this performance issue.
  
  This problem can cause client backup and restore operations
  to take significantly longer than normal.
  
  Customer/L2 Diagnostics:
  The most notable behavior will be an increase in concurrently
  active client sessions activity reported by the QUERY SESSION
  administrative command.  The following diagnostics can then
  be used to confirm this problem:
  
  1. Issue the SHOW THREADS administrative command against the
     Tivoli Storage Manager server every 5 minutes for 15
     minutes.
  2. Look for threads within the output generated that include
     the ICC_RAND_bytes function that end in the
     _global_lock_common function.  For example, there should be
     numerous threads that look similar to the following:
  
      Thread 4122, Parent 70: psSessionThread, Storage 169122,
      AllocCnt 12 HighWaterAmt 174272
        tid=4222, ptid=2922, det=1, zomb=0, join=0, result=0,
        sess=100924
      Stack trace:
        0x09000000007d37b8 _global_lock_common <---------------
        0x09000000007e0c80 _mutex_lock
        0x09000000031a8dac OSSL_locking_callback
        0x0900000003204f88 C98E_CRYPTO_lock
        0x09000000031ad69c efCRYPTO_lock
        0x09000000031dd974 fips_rand_bytes
        0x090000000321ca60 C98E_RAND_bytes
        0x09000000031b4914 METAC_RAND_bytes@AF377_226
        0x090000000129975c ICCC_RAND_bytes@AF323_193
        0x0900000001295110 ICCC_RAND_bytes
        0x090000000128b858 ICC_RAND_bytes      <---------------
        0x000000010014291c GenRandomKey
        .....
        0x000000010000c264 StartThread
  
  3. Each of the above threads will be waiting on a single
     thread, also including the ICC_RAND_bytes function, but
     this time ending with the getbyte() function.  For
     example, there should also be a single thread that looks
     similar to the following:
  
     Thread 1234, Parent 14: psSessionThread, Storage 152138,
     AllocCnt 60 HighWaterAmt 152138
       tid=9014, ptid=2946, det=1, zomb=0, join=0, result=0,
       sess=70983
     Stack trace:
         0x09000000031c9b20 getbyte            <---------------
         0x09000000031c89f8 trng_raw_gen
         0x09000000031c79bc CLiC_trng
         0x09000000031caa60 efGenerateRandomSeed
         0x09000000031c7028 PRNG_GenerateRandomSeed
         0x09000000031c71fc efRNG_ReSeed
         0x09000000031c74d8 OldefRNG_Generate
         0x09000000031c64b8 efRNG_Generate
         0x09000000031dd9b0 fips_rand_bytes
         0x090000000321ca60 C98E_RAND_bytes
         0x09000000031b4914 METAC_RAND_bytes
         0x090000000129975c ICCC_RAND_bytes
         0x0900000001295110 ICCC_RAND_bytes
         0x090000000128b858 ICC_RAND_bytes     <---------------
         0x000000010014291c GenRandomKey
         .....
         0x000000010000c264 StartThread
  
  Initial Impact:
  High
  
  Tivoli Storage Manager Versions Affected:
  All V6.3 Tivoli Storage Manager server users on AIX Power 7.
  
  Additional Keywords:
  ZZ63 gskit getbyte performance perf slow backups client upgrade
  pssessionthread session sessions ICC_RAND_BYTES GenRandomKey
  

Local fix

• An emergency fix is available.  Contact Support for assistance.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users.     *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is currently projected to be fixed   *
  *                 in level 6.3.5. Note that this is            *
  *                 subject to change at the discretion of IBM.  *
  ****************************************************************
  *
  

Problem conclusion

• This problem was fixed.
  
  Affected platforms:  AIX.
  
  See Flash 1647102 for additional information:
  http://www.ibm.com/support/docview.wss?uid=swg21647102
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R63A PSY

     UP