Slow disk storage pool backup with small client transactions.

Answer

The following scenario has been investigated for a customer that was using Content Manager (CM) with Tivoli Storage Manager:

I. DB backup to tape:

a) DB backup (18:03:02 - 18:11:57):
3,410,108 pages in 8min:55secs or throughput of 25.5 MB/s
b) DB backup (19:44.27 - 19:52:17):
3,413,350 pages in 7min:50secs or throughput of 29 MB/s.

II. Stgpool backups to tape:

a) 20:25:09 - 20:38:28 (13min:19secs):
- 90,177 items backed up totaling 4,525,228,032 bytes or avg file size of 49KB
- throughput: 5.5 MB/s.
- This throughput is consistent with normal Content Manager (CM) behavior, which backs up each small file (ranging from 20-60 KB in size) in a separate Tivoli Storage Manager session/transaction.

III. Further verification of the small 'aggregate' size in Tivoli Storage Manager's stgpool volumes:

Excerpt from Tivoli Storage Manager server instrumentation trace file (8/21/06):

Instrumentation began 13:51:04.516 ended 20:34:51.695 elapsed 24227.178

Thread 46 DiskServerThread parent=0 (AIX TID 1146929) 13:51:04.516-->
20:34:51.695
/dev/rtsm_knefile_5
Operation Count Tottime Avgtime Mintime Maxtime InstTput Total KB
--------------------------------------------------------------------
Disk Read 18729 42.462 0.002 0.000 0.086 16994.1 721612
Disk Write 32985 44.379 0.001 0.000 0.974 19464.5 863824
Thread Wait 43153 24139.808 0.559 0.000 31.295
Unknown 0.528
--------------------------------------------------------------------
Total 24227.178 65.4 1585436

It looks like the I/O block sizes for DISK stgpool vols (e.g /dev/rtsm_knefile_5 above) are very small:
Disk Reads: 721612 KB / 18729 (or avg 22 KB/IO)
Disk Writes: 863824 KB / 32985 (or avg 26 KB/IO).


Conclusion:

"Normal" Tivoli Storage Manager client backups, with files backed up in groups (also known as "transactions"), result in data being stored on Tivoli Storage Manager stgpool vols in "aggregates" of 1.5-2.5 MB each. This allows Tivoli Storage Manager server to read & write to stgpool vols using 256KB I/O blocks.

Here each small file is backed up in a separate Tivoli Storage Manager transaction, resulting in small aggregates being stored on Tivoli Storage Manager stgpool vols.
In this case backup stgpool throughput of 5.5 MB/s (avg file size 49KB above) is the best achievable throughput.

Note: