IT47534: VSNAP SERVER DEDUP TABLE GROWS IN SIZE AND CAUSES PERFORMANCE IMPACT AFTER UPGRADE TO 10.1.16 IFIX4.

APAR status

• Closed as fixed if next.

Error description

• After upgrading the IBM Storage Protect Plus vSnap server to
  10.1.16.ifix4, the deduplication table (DDT) on the vSnap server
  starts growing in size
  The DDT size can be observed using the command "vsnap system
  stats". In some cases, the size is displayed as "N/A" in which
  case the size can alternatively be observed using the command
  "zpool status -D".
  
  If the DDT size (in core) exceeds the available ARC size, the
  I/O operations on the vSnap server can be very slow which can
  affect performance of IBM Storage Protect jobs.
  
  When deduplication is enabled on a vSnap server and when many
  terabytes of data has already been written into the vSnap
  storage pool, further I/O operations can be very slow.
  
  This problem is seen with various symptoms, such as:
  
  - All backup operations show very slow throughput.
  - Operations to create target volumes or NFS shares on vSnap
  fail with timeouts.
  - VM backup operations appear to hang/stall for hours.
  - SQL backup operations fail with errors seen in the job log:
  "Failed to mount for volume <name>, Volume id/name is not
  found."
  - Microsoft SQL backup operations fail with errors seen in the
  job log:
  "The system cannot find the file specified, The device is not
  ready."
  
  The issue occurs because the newer version of ZFS included in
  10.1.16 ifix4 contains a different mechanism to manage the DDT
  size, and vSnap does not correctly handle this new mechanism.
  
  |MDVREGR 10.1.6 5737SPLUS|
  
  IBM Storage Protect Plus Versions Affected:
  IBM Storage Protect Plus 10.1.16.4 and later
  

Local fix

• A local patch is now available which can be applied on the
  affected system after upgrading or while upgrading to 10.1.16.4.
  Please contact support for the patch.
  
  Additionally, if the existing size of the DDT already exceeds
  the quota, then run the following command to shrink the DDT.
  This command performs a mass eviction by removing unnecessary
  unique blocks from the DDT.
  "sudo zpool ddtprune -p 100 vpoolX"
  
  After running the ddtprune command, use command "zpool status
  -D" to check the DDT details. The entries with refcnt=1 should
  be purged, and the overall size of the DDT should shrink over
  the next few minutes.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Storage Protect Plus level 10.1.16.4                     *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * see Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  *  Apply the fixing level when available. This problem is      *
  * currently projected to be fixed in IBM Storage Protect Plus  *
  * 10.1.17 level.                                               *
  *  Note that this is subject to change at the discretion of    *
  * IBM.                                                         *
  ****************************************************************
  

Problem conclusion

Temporary fix

•  Apply the eFix on top of 10.1.16.4.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels