Buffer copy elimination and restore and retrieve

Edit online

The IBM® Storage Protect server controls the amount of data to be placed in the buffer, based on tape access optimization with restore and retrieve. This method is not as beneficial to the application as the normal method of getting data. During prototyping, check the performance of the buffer copy elimination method and use this method only if you see a worthwhile improvement.

The maximum amount of data in a single buffer that is returned by the IBM Storage Protect server is (256K bytes – header overhead). As a consequence, only applications that deal with small buffer writes benefit from this data retrieval mechanism. The application must give special attention to the number of bytes in the buffer, depending on the object size, the network, and other boundary conditions. In some situations, the use of buffer copy elimination can actually perform worse than the normal restore. The API normally caches the data and returns a fixed length to the application. The application can then control the number of data writes back to the disk.

If you use buffer copy elimination, create a data-caching mechanism for buffers that are less than the preferred write buffer size. For example, if an application writes 64K data blocks to disk, the application must take these actions:
  1. Call dsmGetBufferData.
  2. Write out blocks of 64K.
  3. On the final block, copy the remainder to a tempBuff, issue another dsmGetBufferData call, and fill the tempBuff with the rest of the data.
  4. Continue writing blocks of 64K:
dsmGetBufferData #1 get 226K        dsmGetBufferData #2 get 240K
Block1 64K - write to disk          Block1 30K - copy to tempbuff-write to disk
Block2 64K - write to disk          Block2 64K - write to disk
Block3 64K - write to disk          Block3 64K - write to disk
Block4 34K - copy to tempbuff       Block4 64K - write to disk
Block5 18K - write to tempbuff        etc

In this example, six disk writes are direct and 1 is cached.

The overall sequence of calls for restore and retrieve is as follows:

dsmInitEx (UseTsmBuffers = True numTsmBuffers = how many buffers the application wants to allocate).

dsmBeginGetData 
While obj id
  dsmGetObj (no data restored on this call- buffer set to NULL)
 While data to read
   dsmGetBufferData (returns the data in the data buffer)
  ...process data... 
  dsmReleaseBuffer  
dsmEndGetObj
dsmEndGetData

For every dsmGetBufferData call, implement a dsmReleaseBuffer call. The dsmGetBufferData and corresponding dsmReleaseBuffer do not need to be consecutive. An application might issue multiple dsmGetBufferData calls first to get several buffers, and then issue the corresponding dsmReleaseBuffer calls later. For sample code that uses this function, see callbuff.c in the API sample directory.

Restriction: Because the API provides the buffer and the goal is to minimize processor utilization, more processing of the data in the buffer is not permitted. The application cannot use encryption and compression with buffer copy elimination because both of these operations require data processing and copies.

Implement both the regular data movement path and the buffer copy elimination to enable the user to switch between both paths, based on their needs. If the user must compress or encrypt data, then use the existing mechanism. If there is a processor constraint, then use the new mechanism. Both of these mechanisms are complementary and do not completely replace each other.