API client-side data deduplication

Edit online

Client-side data deduplication is used by the API on the backup-archive client, to remove redundant data during backup and archive processing before the data is transferred to the IBM® Storage Protect server.

Client-side data deduplication is used by the API, to remove redundant data during backup and archive processing before the data is transferred to the IBM Storage Protect server. Using client-side data deduplication can reduce the amount of data that is sent over a local area network. Using client-side data deduplication can also reduce the IBM Storage Protect server storage space.

When the client is enabled for client-side data deduplication, and you perform a backup or archive operation, the data is sent to the server as extents. The next time a backup or archive operation is performed, the client and server identify which data extents have already been backed up or archived, and send only the unique extents of data to the server.

For client-side data deduplication, the server and API must be at version 6.2 or later.

Before you use client-side data deduplication to back up or archive your files, the system must meet the following requirements:

  • The client must have the deduplication option enabled.
  • The server must enable the client for client-side data deduplication with the DEDUP=CLIENTORSERVER parameter on either the REGISTER NODE or UPDATE NODE command.
  • The storage pool destination for the data must be a data deduplication-enabled storage pool. The data deduplication-enabled storage pool is file device type only.
  • Ensure that the files are bound to the correct management class.
  • A file can be excluded from client-side data deduplication processing. By default, all files are included.
  • Files must be larger than 2 KB.
  • The server can limit the maximum transaction size for data deduplication by setting the CLIENTDEDUPTXNLIMIT option on the server. See the server documentation information about this option.

If any of these requirements are not met, data is processed normally, with no client-side data deduplication.

Here are some data deduplication restrictions:

  • LAN-free data movement and client-side data deduplication are mutually exclusive. If you enable both LAN-free data movement and client-side data deduplication, LAN-free data movement operations complete and client-side data deduplication is ignored.
  • Encryption and client-side data deduplication are mutually exclusive. If you enable both encryption and client-side data deduplication, encryption operations complete and client-side data deduplication is ignored. Encrypted files, and files that are eligible for client-side data deduplication, can be processed in the same operation, but are done in separate transactions.
    Requirements:
    1. In any transaction, all files must be either included for data deduplication or excluded. If the transaction has mixed files, the transaction fails, and a return code of DSM_RC_NEEDTO_ENDTXN is returned by the API.
    2. Use storage device encryption together with client-side data deduplication. Because SSL is used in combination with client-side deduplication, there is no need for client encryption.
  • The following functions are not available for client-side data deduplication:
    • IBM Storage Protect for Space Management (HSM) client
    • API shared buffer
    • NAS
    • Subfile backup
  • Buffer copy elimination cannot be used with data transformations like compression, encryption, and data deduplication.
  • If you use client-side deduplication, the API detects and fails (with RC=254) backups of file extents that are marked as expired on the server during sending data to the server. If you want to retry the operation, you need to include that programming in the calling application.
  • Simultaneous-write operations on the server takes precedence over client-side data deduplication. If simultaneous-write operations are enabled, client-side data deduplication does not occur.
    Restriction: When client side data deduplication is enabled, the API cannot recover from a state where the server has run out of storage on the destination pool, even if there is a next pool defined. A stop reason code of DSM_RS_ABORT_DESTINATION_POOL_CHANGED is returned and the operation fails. There are two ways to recover from this situation:
    1. Ask the administrator to add more scratch volumes to the original filepool.
    2. Retry the operation with data deduplication disabled.

For even greater bandwidth savings, you can enable a local cache for data deduplication. The local cache saves queries from going to the IBM Storage Protect server. The default value for ENABLEDEDUPCACHE is NO, so that the cache is not out of sync with the server. If the cache is out of sync with the server, the application resends all data. If your application can retry on a failed transaction, and you want to use the local cache, set the ENABLEDEDUPCACHE option to YES in the dsm.opt (Windows) or dsm.sys (UNIX) file.

At the end of a restore, if all of the data was restored through the API, and the object was deduplicated by the client, an end-to-end digest is calculated and compared to the value calculated at backup time. If those values do not match, error DSM_RC_DIGEST_VALIDATION_ERROR is returned. If an application receives this error, the data is corrupt. This error can also be a result of a transient error on the network, so try the restore or retrieve again.

Here is an example of the query session command showing data deduplication information:

dsmQuerySessInfo Values:
Server Information:
Server name: SERVER1
Server Host: AVI
Server port: 1500
Server date: 2009/10/6 20:48:51
Server type: Windows
Server version: 6.2.0.0
Server Archive Retention Protection : NO
Client Information:
Client node type: API Test1
Client filespace delimiter: :
Client hl & ll delimiter: \
Client compression: Client determined (3u)
Client archive delete: Client can delete archived objects
Client backup delete: Client CANNOT delete backup objects
Maximum objects in multiple object transactions: 4096
Lan free Enabled: NO
Deduplication : Client Or Server
General session info:
Node: AVI
Owner: 
API Config file:

Here is an example of the query management class command showing data deduplication information:

Policy Information:
Domain name: DEDUP
Policyset name: DEDUP
Policy activation date: 0/0/0 0:0:0
Default management class: DEDUP
Backup retention grace period: 30 days
Archive retention grace period: 365 days
Mgmt. Class 1:
Name: DEDUP
Description: dedup - values like standard
Backup CG Name: STANDARD
Frequency: 0
Ver. Data Exists: 2
Ver. Data Deleted: 1
Retain Extra Ver: 30
Retain Only Ver: 60
Copy Destination: AVIFILEPOOL
Lan free Destination: NO
Deduplicate Data: YES

Archive CG Name: STANDARD
Frequency: 10000
Retain versions: 365
Copy Destination: AVIFILEPOOL
Lan free Destination: NO
Retain Init : CREATE
Retain Minimum : 65534
Deduplicate Data: YES