IT00539: NETWORK ERRORS CAUSING INVALID CALL SEQUENCE ERROR ANS0238E. CAN ALSO CAUSE UNRECOVERABLE DATA MOVER VMWARE BACKUPS.

IBM Tivoli Storage Manager (TSM) client 7.1.0.x interim fix downloads
IBM Tivoli Storage Manager (TSM) Client 7.1.1 Downloads and READMEs

APAR status

• Closed as program error.

Error description

• Network communication errors could be reported as follows in the
  client dsmerror.log :
  
  <timestamp> ANS1017E Session rejected: TCP/IP connection
  failure.
  <timestamp> ANS0238E The sequence of calls is invalid.
  <timestamp> ANS0238E The sequence of calls is invalid.
  <timestamp> ANS0361I DIAG: ..\..\common\vm\vmbackvddk.cpp(5288):
  VmSendData(): Error 2041 creating disk group objects.
  <timestamp> ANS1228E Sending of object '<vmname>' failed.
  
  And in the server activity log only this message will be seen :
  <timestamp> ANE4174E (Session: xxx, Node: <datacenter node>)
  Full VM backup of VMware Virtual Machine '<vmname>' failed with
  RC=2041 mode=Incremental Forever - Full, target node
  name='<datacenter node>', data mover node name='<datamover
  node>'  (SESSION: xxx)
  
  The error ANS0238E should not appear.
  
  In addition, network errors that occur during the last
  transaction of a VMware full backup being performed
  via a TSM data mover can make that backup unrecoverable,
  even though the backup was reported as successful.
  Other errors occurring at that same point can also
  cause the issue, such as server out-of-space
  conditions or server out-of-scratch-volume conditions.
  Non-VMware or In-Guest VMware backups are not affected.
  
  Tivoli Storage Manager Versions Affected:
  Tivoli Storage Manager Client 6.4 and 7.1 Windows and Linux
  
  Customer/L2 Diagnostics
  
  in a Client trace, the following gwill be seen :
  
  <timestamp> ... : vmAPISendData::termSendData(): error in
  dsmEndSendObjEx. rcMsg=ANS1017E (RC-50) Session rejected: TCP/IP
  connection failure.
  ... =========> Entering
      vmAPISendData::beginQuery()
  <timestamp> ... : vmAPISendData::beginQuery(): Performing backup
  query.
  <timestamp> ... : vmAPISendData::beginQuery(): error in
  dsmBeginQuery. rcMsg=ANS0238E (RC2041) The sequence of calls is
  invalid.
  <timestamp> ... : vmVddkQueryVM(): beginQuery() failed with rc =
  2041
  
  Initial Impact: High
  
  Additional Keywords: zz71 network -50 rc-50 rc=-50 firewall
  

Local fix

• Ignore the error ANS0238E and correct the cause of the network
  errors, unless the network errors occurred during the
  end of a full VMware backup by the data mover.
  In that case, re-run the full VMware backup (-mode=full
  or -mode=iffull)
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: Backup-archive client versions 7.1.0, 6.4.0, *
  *                 and 6.4.1 on Windows and Linux x86 systems.  *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION                   *
  ****************************************************************
  * RECOMMENDATION: Apply 7.1.1 or 6.4.2.1 or higher.            *
  ****************************************************************
  A return code was not checked before
  we continue to call other functions for processing.
  

Problem conclusion

• The return code is now checked before
  we continue to call other functions for processing.
  
  Note: while this fix prevents the issue for new full VMware
  backups initiated by a data mover, existing unrecoverable
  full VMware backups and their incrementals will continue
  to be unrecoverable. A new full VMware backup needs to be
  initiated for that VM using the fixed data mover level.
  See IT04201 for detection and full backup initiation options.
  

Temporary fix

• The return code is now checked in the 7.1.0.3 and 6.4.2
  and later deliverables, for new backups, including new
  full VMware data mover backups. However, new options to detect
  unrecoverable VMware backups and initiate a full backup for them
  if desired are not added until 7.1.1 and 6.4.2.1 and later,
  with IT04201.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM CLIENT

• Fixed component ID

  5698ISMCL

Applicable component levels

• R64L PSY

     UP

• R64W PSY

     UP

• R71L PSY

     UP

• R71W PSY

     UP