IT30343: FAILURE DURING OFFLOAD REPORTS FAILED TO CREATE GATEWAY DEVICE: CHECKSUM FAILED

APAR status

• Closed as program error.

Error description

• During an IBM Spectrum Protect Plus offload operation, it can
  fail with the following error messages:
  
      WARN,...,CTGGA1915,Error occured on Storage Vsnap
        replication Please check Vsnap Storage logs.
      ERROR,...,CTGGA0309,Copy failed for snapshot (ID: nnn) from
        source [server: <hostname> volume: spp_<volume name>
        snapshot: spp_<snapshot name>] to
        target [server: <hostname> volume: <volume name>]. Error:
        Exception: Failed to create gateway device: Checksum
        failed for /opt/vsnap-data/offload_<id>/bin_blockmap.
        Expected <ID> was <ID>
      ERROR,...,Skipping remaining snapshots for volume <hostname>
        :spp_<volume name> due to unrecoverable error for Vsnap
        session <session ID>
    This occurs when the checksum of the blockmap metadata cached
    in the offload cache area (/opt/vsnap-data/) does not the
    match expected value recorded in the vSnap configuration DB.
    This checksum mismatch typically occurs due to errors in
    updating the DB that occurred during a previous run of the
    offload job.
  IBM Spectrum Protect Plus Versions Affected:
    10.1.4
  Customer/L2 Diagnostics (If Applicable)
    N/A
  
  Initial Impact:
    Low
  Additional Keywords:
    TS002596475 BLOCKMAP
  

Local fix

•   As a workaround, delete the offload directory using the
    command:
  
      rm -rf /opt/vsnap-data/offload_<id>
  
    Once deleted, try another incremental offload, which will
    download the proper blockmap to the system from the cloud.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.4.                      *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in IBM Spectrum Protect Plus level     *
  * 10.1.5. Note that this is subject to change at the           *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• The problem occurred due to problems with database locking. At
  the end of a successful cloud offload vSnap attempts to update
  some configuration database tables record the status of the
  offload session. This includes storing the checksum of blockmap
  metadata files in the configuration DB. If the database is under
  load during this step, the offload process may fail to save the
  new checksum.
  
  As a result the DB still contains an older checksum which is no
  longer valid. On subsequent offload attempts, vSnap attempts to
  verify the checksum of the metadata against the value stored in
  the DB. The offload fails because the value stored in the DB is
  out of date.
  
  The problem has been resolved by improving the database locking
  process as well as improving the wait-and-retry logic to ensure
  that the configuration DB is correctly updated at the end of a
  successful offload.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels