Duplicate-identification processing can incorrectly deduplicate fragmented data.

Content

Releases affected:
Tivoli Storage Manager Version 7.1.0.xxx and Tivoli Storage Manager V7.1.1.0xx server for all platforms.
All other releases and levels are not affected.

Required Conditions:
During duplicate-identification processing, files larger than the MAXFRAGMENTSIZE option that are fragmented, can be incorrectly processed resulting in fewer bytes being restored than were backed up. The default value for this option is 10G (10000M). These same files that are replicated to a target replication server, contained in a backup set, or contained in an export, might be missing a few of bytes of data. The number of missing bytes is no more than 10 bytes per fragment.

When the client attempts to restore the file, not all bytes are returned due to the invalid length of the first extent in the fragment. This problem results in what appears to be a data integrity issue. The data is complete in the storage pool on the server and can still be restored by moving the data to a non-deduplicated sequential storage pool.

There are a few situations where a data integrity issue can occur:

1) If a backup set is created that includes a file that contains fragments that were deduplicated incorrectly, the file will be missing some data when the file is restored from the backup set.

2) If an export is performed that includes a file that contains fragments that were deduplicated incorrectly, the file will be missing some data when the file is imported from the export media.

3) If node replication is performed that includes a file that contains fragments that were deduplicated incorrectly, the file might be missing data on the target replication server.

To issue the following select statements you must connect to the DB2 instance for the Tivoli Storage Manager server.

In a Windows environment, complete the following steps :
1) Open a Windows command prompt.
2) Run the db2cmd command. This opens a new command window with the DB2 environment set up.
3) From this new window, run the following commands :
set DB2INSTANCE=SERVER1
db2 connect to TSMDB1
NOTE: If you have more than one server instance then change the DB2INSTANCE environment variable to select the correct instance.

In a UNIX or Linux environment, complete the following steps :
1) Log on with the instance user ID.
2) From a shell, run the following command :
db2 connect to TSMDB1

After you connect to the database, you can run DB2 select commands.

Issue the following SQL statement on the source server to determine whether fragmented files were incorrectly deduplicated:

db2 -x " select distinct 'show invo ' || bfsa.objid from tsmdb1.bf_super_aggregates bfsa left join tsmdb1.bf_aggregated_bitfiles bfbf on (bfsa.bfid=bfbf.superbfid ) left join tsmdb1.af_bitfiles afbf on ( bfbf.srvid=afbf.srvid and bfbf.superbfid=afbf.bfid ) where bfbf.offset=0 and bfbf.length>23 and bfsa.sequence>0 and afbf.flags not in ( 1, 3, 5, 7, 9 )" > showinvo.mac

Once the DB2 command completes, run the showinvo.mac macro from the administrator console. Provide the output from the macro to your IBM service representative. The output from this macro includes information about each file that is affected by this problem.

If any of these files were included in a backup set or in an export, that file will not be able to be restored completely from that backup set or that export media. After the fix for APAR IT05283 is applied, a new backup set or export can be performed that will contain the data for this file.

If any of these files were included in a node replication, the file needs to be deleted from the target server and node replication processing must be performed again. The following statement can be issued on a source replication server to identify affected files that were replicated to a target replication server:

db2 "select nrro.repl_objid, nrrs.servername, nrro.objid from tsmdb1.replicated_objects nrro left join tsmdb1.replicating_servers nrrs on nrro.serverid=nrrs.serverid where nrro.objid in ( select bfsa.objid from tsmdb1.bf_super_aggregates bfsa, tsmdb1.bf_aggregated_bitfiles bfbf, tsmdb1.af_bitfiles afbf where bfbf.srvid=afbf.srvid and bfbf.superbfid=afbf.bfid and bfsa.bfid=bfbf.superbfid and bfbf.offset=0 and bfbf.length>23 and bfsa.sequence>0 and afbf.flags not in ( 1, 3, 5, 7, 9 ) )"

The following shows an example of the output from this command:

REPL_OBJID      SERVERNAME          OBJID
----------      ----------      ---------
      5940      SERVER2              1026
      5945      SERVER2              8201

2 record(s) selected.

The OBJID column displays the object id on the source replication server. This object id will match the output from the previous select statement that created the showinvo.mac file.

The SERVERNAME column displays the name of the server that this file is replicated to.

The REPL_OBJID column displays the object id on the target replication server.

To delete the file from the target replication server, issue the following command on SERVER2 to delete this object:

DELETE OBJECT 5940

This object will be replicated again to the target server during the next node replication process.

Before you issue the DELETE command, you can verify that you are deleting the correct file on the target replication server. For example, issue the command SHOW INVO 1026 on the source replication server and SHOW INVO 5940 on the target replication server. Verify that they are the same object before you issue the command DELETE OBJECT 5940 on the target replication server.

Contact IBM support if assistance is required in performing the previous steps.

Circumvention:

You can move the data to a non-deduplicated sequential storage pool, which will allow the data to be restored. To move the data, follow these steps:
1) For each SUPERBFID from the previous SQL statement, issue the following command:
SHOW BFO superbfid
2) The volume that the superbfid resides on will be displayed at the end of the SHOW BFO command output. The output might show more than one volume, however the MOVE DATA command needs to be issued for only one of the volumes that the file is stored on.
3) Issue the MOVE DATA command to move the data on the volume to a non-deduplicated sequential storage pool (NOTE: It must be a sequential storage pool not a random access type of storage pool.):
MOVE DATA volume_name STGPOOL=non-deduplicated_sequential_stg_pool
4)Try the restore again.

The fix for APAR IT05283 does not correct these fragmented headers in the server. However, the fix for the APAR prevents new files from being created with these incorrect fragment headers. In addition, it detects if there is a file with incorrect fragment headers and allows the data to be restored, replicated, exported, and generated in a backup set properly when this condition exists. The fix for this APAR must be applied to both the source and the target replication servers to fix the problem for node replication.

Problem Resolution:
APAR IT05283 was created for this problem. Apply the fixing level when available. These problems are fixed in Tivoli Storage Manager server level 7.1.1.100, 7.1.2 and above for all server platforms.