Scenario: Improving performance of client backups

The scenario illustrates how to interpret client instrumentation reports to resolve a problem with backup operations.

Problem

Dave, an IBM Storage Protect administrator, is experiencing a performance degradation of backup operations on a file server. He is backing up the file server from an AIX® backup-archive client to an AIX server, and the performance has degraded from an average throughput of 32 MB per second to 15 MB per second in the last few days.

Goal

Dave expects to restore his previous throughput level during backups.

Data collection

Dave collects the following data:

Client instrumentation data
Server monitoring data, which is generated by the servermon component

Analysis and determination of the bottleneck

During the backup operation, the data flows from the client, over the network, to the IBM Storage Protect server. The data is transferred from the disk system through a host bus adapter (HBA) that connects the disk to the IBM Storage Protect client. The client backs up the data to the server through a local area network (LAN) connection. A Network Interface Card (NIC) connects the client to the LAN and a separate NIC connects the LAN to the server. The data is backed up from the server to disk and to a tape library through separate HBA devices.

The image shows the path that the data takes during a backup operation. The data flow transfers the data from the disk through a host bus adapter (HBA) that connects the disk to the IBM Storage Protect client. The client backs up the data to the server through a local area network (LAN) connection. A Network Interface Card (NIC) connects the client to the LAN and a separate NIC connects the LAN to the server. The data is backed up from the server to disk and to a tape library through separate HBA devices. — Figure 1. Data flow for client backup operations over a local area network

Following the flow of data, Dave looks at the client data first.

The most recent backup operation finished with the following statistics:

Total number of objects inspected:        1
Total number of objects backed up:        1
...
Total number of bytes transferred:    11.80 GB
LanFree data bytes:                   11.80 GB
Server-Free data bytes:                   0  B
Data transfer time:                  216.01 sec
Network data transfer rate:        57,294.91 KB/sec
Aggregate data transfer rate:      16,542.69 KB/sec
Elapsed processing time:           00:12:28
Average file size:                 11.66 GB

To determine where the slowdown occurs, Dave runs a test backup of the file server with the testflag=instrument:detail option. By setting this option, Dave collects client instrumentation data about the backup operation. He reviews the instrumentation output. The client instrumentation report shows that, during the backup, a large amount of time was spent on File I/O operations. The elapsed time of the backup was 746.666 seconds, and 524.380 seconds were spent in File I/O. By looking at the client instrumentation categories in Table 1, Dave knows that the File I/O category represents the elapsed time for file read and write operations.

Thread: 2571  Elapsed time   746.666 sec
Section      Actual (sec) Average(msec)  Frequency used
------------------------------------------------------------------
Process Dirs         0.000           0.0              0
Solve Tree           0.000           0.0              0
Compute              0.234           0.0          48345
BeginTxn Verb        0.000           0.1              2
Transaction          0.715         357.5              2
File I/O           524.380          10.8          48346
Compression          0.000           0.0              0
Encryption           0.000           0.0              0
CRC                128.042           2.6          48398
Delta                0.000           0.0              0
Data Verb           87.912           1.8          48345
Confirm Verb         0.136           8.5             16
EndTxn Verb          2.234        1117.0              2
Other                4.513           0.0              0
------------------------------------------------------------------

The rate of data transfer is calculated by dividing the file size by the time that is recorded in the File I/O field in the instrumentation output:

(11.66 GB x 1024 MB/GB)/524.380s = 22.8 MB/s

The backup reads the data only from the client disk, so Dave suspects a problem with the disk on the client system.

Problem resolution

Upon further investigation, Dave discovers that the AIX file system was recently mounted with the cio option, which enabled concurrent I/O in the file system. He concludes that mounting the file system with the AIX concurrent I/O option caused a degradation in backup performance. Concurrent I/O prevents file system read-ahead operations.

Dave modified the system settings to ensure that the AIX file system is not mounted with the cio option. As a result, the performance of backup operations is restored to its previous level.