IT13794: IMPROVE TPC HYPERVISOR PROBE BEHAVIOR IN ERROR CASES

IBM Spectrum Control V5.2.10 (May 2016)
IBM Spectrum Control V5.2.10.1 (July 2016)
IBM Spectrum Control V5.2.11 (August 2016)
IBM Spectrum Control V5.2.12 (November 2016)
IBM Spectrum Control V5.2.13 (March 2017)
IBM Spectrum Control V5.2.14 (May 2017)
IBM Spectrum Control V5.2.15 (August 2017)
IBM Spectrum Control V5.2.15.2 (November 2017)
IBM Spectrum Control V5.2.16 (March 2018)
IBM Spectrum Control V5.2.17 (May 2018)

APAR status

• Closed as program error.

Error description

• Lack of Datastore browse permission will fail the probe without
  informative log messages.
  Timeout errors in vSpheres API produce unexpected behaviour in
  TPC ESX probes and discoveries.
  
  TPC code needs to be improved to better handle the vCenter API
  calls: retrieveDiskPartitionInfo
  and    searchDatastoreSubFoldersTask
  
  The two vCenter API calls:
  -  retrieveDiskPartitionInfo times out, or takes too long.
  
  In the first (timeout) case the whole probe or discovery fails
  and leaves the VIClientWrapper pooled object in inconsistent
  state which affects subsequent vCenter calls which might fail
  with "javax.xml.ws.soap.SOAPFaultException: The session is not
  authenticated." or " javax.xml.ws.soap.SOAPFaultException:
  Unable to communicate with the remote host, since it is
  disconnected."
  
  In the latter case the whole probe takes very much and delays
  other ESX probes running in parallel.
  
  Usually increasing the timeout for this call is not a good idea,
  because it will only prolong the whole probe running time.
  
  - searchDatastoreSubFoldersTask fails because of incorrect
  privileges with little info provided in the probe job log.
  
  The probe job log should contain a clear message about why
  the call failed and on what Data Store.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * SC users with hypervisors.                                   *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * Following exception appear in                                *
  * device/log/tpc/traceTPCDeviceServer*:                        *
  *                                                              *
  * javax.xml.ws.soap.SOAPFaultException: Permission to perform  *
  * this operation was denied.                                   *
  *         at                                                   *
  * com.sun.proxy.$Proxy.searchDatastoreSubFoldersTask(Unknown   *
  * Source)                                                      *
  *         at java.lang.reflect.Method.invoke(Method.java:620)  *
  *         at                                                   *
  * com.ibm.tpc.vmmgr.client.VIClientWrapper.performMethodInvoca *
  * tion(VIClientWrapper.java:510)                               *
  *                                                              *
  * ESX Statistics Discovery failed with exception.              *
  * java.lang.Exception:                                         *
  * com.ibm.tpc.vmmgr.client.VITimeoutException: Operation timed *
  * out.                                                         *
  *         at                                                   *
  * com.ibm.tpc.vmmgr.collection.VMMgrDiscoverESXHypervisorStati *
  * sticsProcess.process(VMMgrDiscoverESXHypervisorStatisticsPro *
  * cess.java:238)                                               *
  * ...............................                              *
  * Caused by: com.ibm.tpc.vmmgr.client.VITimeoutException:      *
  * Operation timed out.                                         *
  *         at                                                   *
  * com.ibm.tpc.vmmgr.client.VIClientWrapperProxy.invoke(VIClien *
  * tWrapperProxy.java:209)                                      *
  *         at                                                   *
  * com.sun.proxy.$Proxy.performMethodInvocation(Unknown Source) *
  *         at                                                   *
  * com.ibm.tpc.vmmgr.scanner.MethodInvocationOperation.performO *
  * peration(MethodInvocationOperation.java:104)                 *
  *                                                              *
  * ESX logical storage probe failed java.lang.Exception:        *
  * com.ibm.tpc.vmmgr.client.VINotConnectedException             *
  * ......................................                       *
  * Caused by: com.ibm.tpc.vmmgr.client.VINotConnectedException  *
  *         at                                                   *
  * com.ibm.tpc.vmmgr.client.VIClientWrapper.collectProperties(V *
  * IClientWrapper.java:420)                                     *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fix maintenance when available.                        *
  ****************************************************************
  

Problem conclusion

• This APAR performs following improvements in how VMWare API
  faults are handled.
   - when the user does not have the required privilege to run
  searchDatastoreSubFoldersTask the following message will be
  displayed in the hypervisor's probe log file:
      "The user <VC User Name> does not have the privilege to
  browse the datastore <Datastore name>"
   - the retrieveDiskPartitionInfo will no longer be called during
  the ESX discovery phase
   - after a VITimeoutException the VIClientWrapper object,
  through which the VMWare APIs are called, will not be reused (
  to prevent VINotConnectedException )
  
  The fix for this APAR is targeted for the following maintenance
  package:
  
  | refresh pack | 5.2-TIV-TPC-RP0011 - target 3Q16
  
  Fixed in IBM Spectrum Control 5.2.11.
  
  http://www.ibm.com/support/docview.wss?&uid=swg21320822
  
  The target dates for future refresh packs do not represent a
  formal commitment by IBM. The dates are subject to change
  without notice.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• Device
  

Fix information

• Fixed component name

  TPC

• Fixed component ID

  5608TPC00

Applicable component levels

• R527 PSY

     UP

• R528 PSY

     UP

• R529 PSY

     UP

• R520 PSY

     UP