Guidelines for Node Replication

Answer

Overview for Implementing Node Replication
A successful implementation of node replication starts with how well you plan the initial replication of data and the subsequent incremental daily replications during normal operations. You must ensure you have dedicated the appropriate, well tuned hardware resource to the task. This document explores these topics in detail and provides you guidance in successfully implementing node replication.

It is important to understand that node replication requires additional resources above what is typically required for a Tivoli Storage Manager (TSM ) 6.3 server. A successful implementation relies on sufficient, dedicated hardware resources. Node replication requires increased amounts of memory and CPU. The database and its logs must be appropriately sized to ensure transactions can complete. A dedicated network, with enough bandwidth to handle the amount of data you intend to replicate, is required. If you are not able to provide sufficient resources to node replication, then you may want to consider the more traditional disaster recovery methods, such as Export/Import or keeping copy storage pool volumes off-site. Or you may want to consider a combination of replication for critical, high priority data and one of the traditional methods.

CPU and Memory Requirements for Node Replication
Node replication processing requires additional memory above what is typically recommended for a TSM 6.3 server. Running a server with insufficient memory may cause server performance to decrease or other operational issues. When running node replication without deduplication, it is recommended you install a minimum of 64 GB of memory. For servers running node replication in conjunction with deduplication, you need a minimum of 128 GB of memory to handle the workload. For larger workloads, you will need higher amounts of memory.

Here are the source server recommended combinations of CPU/RAM for running replication:

• Without deduplication, at least 4 CPU cores, 64B RAM
• With deduplication enabled, at least 8 CPU cores, 128GB RAM

Database Requirements for Node Replication
Node replication requires additional TSM database space to track the files that have been replicated. It is important that you ensure your database is properly sized to accommodate the required space. To determine if your database can handle the additional space requirements, you must first estimate how much additional database space node replication will consume. Issue the QUERY OCCUPANCY command for each node and data type that you plan to replicate. The output from the command contains the number of files for the node and data type. Use the total number of files from all nodes and data types and multiply it by 300 (the number of additional bytes needed for each replicated file) to determine the additional database space needed. If the additional required space approaches or exceeds the size of your database you must increase the available database space. We recommend that you increase the size of your database by the additional amount needed plus an additional 10% of the current database size. Ensure you examine both replication servers and their databases and increase the database size if necessary.

When you create the active log, you need at least 64GB of memory to run replication. If replication and deduplication are both being used, create an active log of 128GB in size.

See the following article to determine if your database and recovery logs are properly sized for Tivoli Storage Manager V6 servers: http://www-01.ibm.com/support/docview.wss?uid=swg21421060

Important: The database and database logs need to be placed on high availability, robust disk that has high performance capability. The database and logs need to be on their own unique directory or mount point. Do not use the same disk or mount point for the database and logs. Similarly, do not use the same disk or mount points for system tasks such as system paging or for other applications as those used for the database and logs. See the following article for further instructions on how to configure the database and logs on disk:
http://www-01.ibm.com/support/docview.wss?uid=swg21394339

Planning the Initial Replication
An initial replication of data is needed to prime the target server with data that exists on the source server. To ensure a successful initial replication you must determine if you have the network bandwidth, CPU and time available for replication given the amount of data that must be replicated. This section guides you through planning for the initial replication.

Estimating Network Bandwidth Required for Replication
Before you proceed with testing replication throughput, ensure that your network is capable of handling the additional replication traffic. You need to following information to calculate the required network bandwidth:

1. Total data (TD) to be replicated in gigabytes. See "Estimating the initial amount of data to be replicated" section for instructions on obtaining the value.
2. Length of desired replication window time (RWT) in hours. This is the time that you allot during your server maintenance window for replication to occur.
3. Deduplication ratio (DR), if using deduplication. Determine DR by inspecting the deduplication enabled storage pool with QUERY STGPOOL FORMAT=DETAIL. If you're not using deduplication, use 100 for DR.

The following formula provided will help with determining the capabilities of your existing network. When you calculate the values, it is best to add generous padding to the numbers used to allow for data growth and periodic peaks to data that is backed up and then replicated. When the total data to be replicated, RWT, and DR are collected you calculate the bandwidth needed with the following formula:
Required Network Bandwidth (Mbits/second) = (TD * ( 100 / DR ) * 8192) / ( RWT * 3600 )

Required Network Bandwidth is the minimum bandwidth needed. If the Required Network Bandwidth value exceeds the capabilities of your current network adjust the values used in the formula. By reducing the total data to be replicated or increasing the replication window the needed bandwidth can be reduced. If you can not adjust the TD or RWT values, adjust or replace you existing network to reduce the additional workload.

Testing Replication Throughput Capability
You must determine how much data you can replicate to the target server given the network and replication servers capabilities. How you 'prime' the target server with replicated data is determined by one question: "Do you have enough network bandwidth and CPU to complete the initial replication in a reasonable amount of time?" To answer this question follow the steps provided below. When you have an answer, choose a method provided below to prime the target server.

Note: Throughput numbers seen in the TSM server test environment indicate that it takes approximately 4 hours to replicate 1.3TB of data; or approximately 325GB/hour. This was accomplished with the following hardware architecture:
The source server is an IBM 3850, CPU: 6 x 4 core Intel(R) Xeon(R) CPU E7450 @ 2.40GHz, MAXSESSION=30. The target server is also an IBM 3850, CPU: 16 x 4 core, Inter(R) Xeon(R) CPU L7555 @ 1.87GHz. The network between the servers is a 10 GB Fiber Channel Over Ethernet (FCOE).
The servers in this test environment are located in the same building. The throughput you experience may decrease due to latency in the network as the distance between the servers increases.

Your first step is to determine how your servers and network performs during replication at various points during your server schedules. A controlled test of replication indicates how many bytes each hour can be replicated over your network during a given time of day. Based on the test results and the total amount of node data that you replicate, you can choose a method for the initial replication.

Important notes:

• To obtain a nominal performance value, repeat the test daily for several days or until you are comfortable that you have determined the nominal throughput value.
• Do not perform the test using deduplication enabled storage pools. Deduplication enabled storage pools can improve the performance of replication but using them during this test might result in inaccurate performance numbers.
• If you have more than one source server replicating to a single target server over the same network and they are performing replication simultaneously then perform this test with all source servers replicating concurrently. However, if you are scheduling the replications at different times, run this test individually from each server. Either way, at the end of this test, each source server must have its own performance number in order to determine which method is best suited for that server.
• If you have two servers that replicate to each other, perform the test while both servers are doing so.

Complete the following steps to determine how much data you can replicate during a specified window:

1. Select one or more nodes and file spaces that have approximately 500 GB to 1 TB of total data. Ensure that the data is typical of the data that you replicate on a routine basis. Ensure that nodes are configured for replication.
2. Select a window of time during which replication normally runs so that you experience the same network constraints that you have during production if the link is shared.
3. If you plan to use SSL during replication, ensure SSL is enabled. Be aware that using SSL impacts the performance of replication and cause it to run slower.
4. Issue the REPLICATE NODE command to start the replication process.
5. After the replication process completes, review the summary message ANR0327I. Use the values "Amount replicated" and "Elapsed time" to determine the number of bytes per hour that can be replicated. For example, the following output indicates that it took 3 hours and 46 minutes to replicate 20.481 GB. Dividing the amount of data replicated by the time it took to replicate it yields the gigabytes an hour that were replicated. You will use this information to determine how much time is needed to perform the initial replication and daily incremental replications.
   

07/22/2013 09:21:50 ANR0327I Replication of node VM01WINT-ARCH completed.Files current: 492,156. Files replicated:164,052 of 164,052. Files updated: 164,052 of 164,052.Files deleted: 197,974 of 197,974. Amount replicated:
20,481 GB of 20,481 GB. Amount transferred: 3,204 GB. Elapsed time: 0 Day(s), 3 Hour(s), 46 Minute(s).(SESSION: 28, PROCESS: 21) 20,481,000,000,000 bytes / 226 minutes =90,623,893,805 bytes per minute * 60 minutes = 5,437,433,628,318 bytes per hour. In this example, the system is capable of replicating approximately 5.5 TB an hour.

After running the controlled replication test, your network may be capable of handling additional workload. To see how your network handles additional workload during replication, increase the value of the MAXSESSIONS parameter on the REPLICATE NODE command and run the test again. Increasing the number of replication sessions allows more data to transfer concurrently during replication. On the other hand, if you determine that 10 replication sessions (the default MAXSESSIONS value) causes your network to degrade below acceptable levels, then decrease the value of the MAXSESSIONS parameter. By adjusting the value of the MAXSESSIONS parameter during a different test, you eventually reach peak network utilization and, therefore, your optimal data transfer capability.

Estimating the total amount of data to be replicated
Because initial replications typically take longer to complete than incremental daily replications, it is important to determine the amount of data that is replicated initially and incrementally daily. Use the following explanations to determine each amount.

Estimating the initial amount of data to be replicated
After you determine how your servers and network will perform during replication, decide which nodes, file spaces and data types will be replicated. Using the QUERY OCCUPANCY command for each node to be replicated, calculate the total amount of physical space occupied for each file space and data type to be replicated. Remember, replication can be tuned to the data type. If you do not plan on replicating a particular data type in a file space, ensure you discount the number of files for that data type. Also, be aware that the physical space occupied is valid for non-deduplication enabled storage pools only. For deduplication enabled storage pools, use the Logical Space Occupied field for your calculations.

Estimating the amount of data to be replicated incrementally daily
To estimate the amount of data that needs to be replicated daily, you must determine the amount of data that is backed up daily by the client nodes will be replicated. For the best results, average the amount of data backed up over several days.

When client nodes complete a store operation, the client logs completion messages with the server. The completion messages report statistics or information for a client operation performed to the server. One of these messages is ANE4961I which reports the number of bytes transferred for this operation. Use the amount reported in the ANE4961I message and average it over several days to find the average amount backed up daily by a particular node. Combine the average amount of all nodes to be replicated to determine how much data is replicated daily.

Calculating the Amount of Time Needed for Replication
When you know the amount of data to replicate and the bytes per hour the network is capable of, estimate how many hours it takes to perform the replication. Use the following formula to determine the time needed for replication:
Hours to Complete Replication = Total bytes to be Replicated / Bytes Per Hour

For the initial replication, you can determine how many hours it takes for the replication to occur over the network during a daily window for replication. Use the following formula to determine this:
Days To Complete Replication = Hours to Complete Replication / 24

Use the value to determine which method you use to complete the initial replication. See the section Selecting a method for the initial replication.

For daily incremental replications, the Hours to Complete Replication value must be larger than the window of time you have allotted for replication. See the section labeled "Planning for Incremental Replication" for guidance on how to tune the replication if needed.

Insufficient Time to Replicate the Amount of Data?
If the time needed to replicate the amount of data is insufficient and you have tuned your network and system to its maximum capacity you must reduce the amount of data that you determine need to be replicated. Re-examine each node and data type and ensure that it must be replicated. Eliminate any nodes where it is not critical that their data be replicated. When you reduce the amount of data being replicated, re-calculate the amount of time needed for replication. Repeat this process until you can complete the replication in the allotted window.

Selecting a method for the initial replication
Based on the results of the test replication and the total amount of data that you want to replicate, determine which of the following methods to use for the initial replication:

Method 1: Export/Import using Tape and Node Replication Synchronization
Use this method if you have a large amount of data to replicate and you cannot wait for initial replication to complete. Export the data for the nodes you intend to replicate and import it on the target server. Be sure to use the DATES=ABSOLUTE parameter when importing the data. Next configure nodes on the source and target servers so that, during replication, the data that was exported from the source server and imported to the target server is "synchronized". Achieve this by updating the node definition on the source server to REPLMODE=SYNCSEND and the node definition on the target server to REPLMODE=SYNCRECEIVE.

When replication is started no data is transferred because the nodes are configured for synchronization. The servers exchange information about the data and add entries to the database to make the data appear replicated. After the synchronization completes, the nodes are automatically configured by Tivoli Storage Manager to replicate their data normally. The node definitions will reflect REPLMODE=SEND on the source server and REPLMODE=RECEIVE on the target server.

Important: Ensure that the DATES=ABSOLUTE parameter is specified when the data is imported to the target server The ABSOLUTE value sets the insertion dates for the objects on the target server to be the same as those on the source server. When synchronization occurs, the servers use these dates to identify files that are the same and updates them to be replicated files.

Method 2: Subset of Nodes Replication
Use this method if you have time for the initial replication to complete but your network cannot handle the workload of replicating all nodes at one time. The goal of this method is to replicate subsets of nodes incrementally based on specific characteristics of the data of the nodes. You can use any characteristic to create subsets of nodes.

When deciding how many nodes to add to a group consider the amount of data that is ingested daily by the nodes and not just the existing amount of data currently on the source server. However, when considering the order of replication, prioritize the subset of nodes that have mission-critical data and perform replication for them first. After these high-priority nodes complete their initial replication, continue to replicate them daily and incrementally add the replication of other subsets of nodes whose data is important, but not mission-critical. Repeat this process until all subsets of all nodes that must be replicated complete their initial replication.

Method 3: Active Data First Replication
Generally, active versions of backup data are considered to be more important than the inactive versions. By setting up an initial replication of only active data, you can avoid the overloading of your network or servers had you replicated both active and inactive data. Configure your replication rules to replicate active data first. After the initial replication of the active data completes, configure the replication rules to replicate all versions of the data. During the next scheduled replication, any new active versions, including all inactive versions, will be replicated. The files that were active but are now inactive are not replicated again.

Method 4: Complete Replication of all Nodes
If you can wait for the initial replication process to complete, finish configuring the nodes and begin or schedule a replication. To ensure that the replication process proceeds as expected, monitor the progress of the replication by issuing a QUERY PROCESS command. Summary information in the output includes the amount of data replicated and the process duration. Use the summary information statistics to determine whether the values of the controlled test match the actual replication values. If the values match, continue the replication. If the values do not match and you are concerned about the amount of time that replication is taking, consider one of the following replication methods.


Planning for Incremental Replication
This section describes guidelines for performing replication on a regular basis, after the initial replication has been completed.

Scheduling Subsequent Incremental Replication
After you complete the initial replication, subsequent incremental replications must be frequent, preferably daily, to ensure that the data on the target server is maintained at an acceptable recovery point.

Daily incremental replications typically do not require as much time to complete as the initial replication. However, if the initial replication took four days to complete, and during that time you were ingesting data daily, your first scheduled replication might require more time than you originally planned for daily incremental replications. However, successive replications will eventually catch up with the data ingested during the initial replication, and you should be able to maintain your replication schedule within the allotted window. If you determine that the replication process is not catching up, consider increasing the number of sessions that are transferring data to the target server at least temporarily until the replications are handling the data ingested each day. Then reduce the number of sessions.

When should you run daily incremental replications? Do not run replication during client backup windows because the data that is replicated is stored at the same time. Wait until after the client backup completes and run replication during the window allotted for server maintenance. You should also avoid running replication at the same time that expiration is running.

Consider replication as a peer process for storage pool backups and schedule replication accordingly. If you are replicating from deduplication enabled storage pools, run processes in the following order (waiting for each to complete before proceeding to the next):

1. Identify Duplicates: This process breaks files into extents, potentially reducing the amount of data to be sent to the target server when replication occurs.
2. Replication: Only file extents that do not exist on the target server are sent during replication, reducing the required bandwidth and improving performance. Replication performance improves as more deduplicated data is stored on the target server. When more extents are stored on the target server, the probability that a duplicate will be found for an extent increases.
3. Reclamation: Reclamation removes and links duplicated extents. Running reclamation after replication completes improves performance because replication does not need access additional linked extents.

For details about scheduling replication along with other server processes, see the following Technote: http://www-01.ibm.com/support/docview.wss?uid=swg21419733

Replication Mount-point Usage
The REPLICATE NODE commands allows the administrator to specify the maximum allowable number of data sessions to use for sending data to a target replication server by specifying the MAXSESSIONS parameter. The default value is 10 and increasing the number of sessions can improve node replication throughput.

When setting the MAXSESSIONS value, consider the number of logical and physical drives that can be dedicated to the replication process. To access a sequential-access volume, Tivoli Storage Manager uses a mount point and, if the device type is not FILE, a physical drive. The number of available mount points and drives depends on the following factors:

• Other Tivoli Storage Manager and system activity
• The mount limits of the device classes for the sequential access storage pools that are involved

Ensure that there are sufficient mount points and drives available to allow node replication processes to complete. Each replication session might need a mount point on the source and target replication servers for storage pool volumes. If the device type is not FILE, each session might also need a drive on both the source and target replication servers.

When setting a value for MAXSESSIONS on the REPLICATE NODE command, consider the available bandwidth and the processor capacity of the source and target replication servers.

Tip:

• The value specified by the MAXSESSIONS parameter applies only to data sessions. Data sessions are sessions during which data is sent to a target replication server. However, if you issue a QUERY SESSION command, the total number of sessions might exceed the number of data sessions. The difference is due to short control sessions that are used for querying and setting up replication operations.
• Remember that the value of the MAXSESSIONS parameter represents the maximum allowable number of sessions and that the number of sessions that are used for replication depends on the amount of data to be replicated. If you are replicating only a small amount of data, there is no benefit in increasing the number of sessions. The total number of sessions might be less than the value that is specified by the MAXSESSIONS parameter.

Deduplication and Replication Mount-point Usage
Replicating data using deduplication enabled storage pools can significantly improve the performance by not having to send extents (chunks) of data that already exist on the target server. For chunks that must be sent, the source server may have to mount several volumes to read the file if it contains chunks that are linked to other chunks on different volumes. To reduce the mounting and dismounting of volumes, deduplication enabled storage pools are allowed to keep more than one FILE volume open and mounted. The number of open and mounted FILE volumes is controlled by the NUMOPENVOLSALLOWED server option. The default value for this option is 10. A value of 10 specifies that session or process reading data from FILE volumes in a deduplication enabled storage pool can keep 10 volumes open and mounted. If more volumes are needed to obtain additional extents, the least recently-accessed volume and its mount point are released. The volume with the required extent is mounted on the just-released mount point.

To avoid replication sessions waiting for a mount point in deduplication enabled storage pools, you may need to adjust the number of mount points allowed. A low mount limit might cause replication sessions already holding mount points to needlessly wait for additional mount points. For example, suppose that the mount limit is set to 10 and that all sessions already have a meta-data volume mounted and that each session needs to mount another volume. In these circumstances, the sessions will stall while waiting for a mount point that is not available. Other sessions or server processes that are using mount points in the device class can also aggravate this problem.

To avoid having replication sessions wait for a mount point in deduplication-enabled storage pools, use the following formula to set the mount limit in the device class to which the deduplication enabled storage pool is assigned:
NUMOPENVOLSALLOWED * MAXSESSIONS

The server option NUMOPENVOLSALLOWED specifies the number of input FILE volumes in a deduplicated storage pool that can be open at one time. The default value is 10. You specify the MAXSESSIONS parameter on the REPLICATE NODE command. The default value for the parameter is 10. If you use the defaults for both values, then update the mount limit in the device class to 100. This value should be sufficient to allow replication and other sessions or processes to acquire mount points. Be aware, however, that running other sessions and processes that use mount points from the same device class as the replication process will use mount points and may affect replication performance.

If your mount limit is already set to a value that is enough to satisfy the number of mount points needed for replication then no action is necessary.

Maintaining the Tivoli Storage Manager Server Service Level
As with any major function, it is important to maintain the Tivoli Storage Manager server at the most recent level of service when you run node replication.

Conclusion
An initial replication primes the target server with data from the source server. A controlled test will indicate how many bytes each hour can be replicated over your network during a given time. Using the test results and a calculation of the total amount of data to be replicated during normal operations, you can select a method for the initial replication. Follow the initial replication with daily, scheduled incremental replications to maintain an acceptable recovery point objective.

In summary, ensure the following for a successful implementation of replication:

• Optimally configured disk, memory, cpu and network
• Optimally sized active and archive logs
• Database sized to account for increased space due to replication. The source and target servers should be the same size.
• Replication is scheduled appropriately. Avoid running during expiration.
• Sufficient number of mount points to avoid stalled replication sessions and other server processes.
• Tivoli Storage Manager 6.3.1 or later.