In today's highly competitive marketplace, it is important to deploy a data processing architecture that not only meets your immediate tactical needs but that also provides the flexibility to grow and change to adapt to your future strategic requirements. In December 2009, IBM introduced the DB2 pureScale Feature for Enterprise Server Edition (DB2 pureScale Feature). The DB2 pureScale Feature leverages an active-active shared-disk database implementation based on the DB2 for z/OS data-sharing architecture. It leverages proven technology from the DB2 database software on the mainframe to bring the active-active shared-disk technology to open systems.
The DB2 pureScale Feature meets the needs of many customers by providing the following key benefits:
- Virtually unlimited capacity
- The DB2 pureScale Feature provides practically unlimited capacity by enabling the addition and removal of members on demand. The DB2 pureScale Feature can scale to 128 members. It has a highly efficient, centralized management facility that allows for very efficient scale out capabilities. The DB2 pureScale Feature also leverages a technology called Remote Direct Memory Access (RDMA) that provides a highly efficient inter-node communication mechanism to augment its scaling capabilities
- Application transparency
- An application that runs in a DB2 pureScale environment does not need to have any knowledge of the different members in the cluster or of partitioning data. The DB2 pureScale Feature automatically routes applications to the members deemed most appropriate. The DB2 pureScale Feature also provides native support for syntax that other database vendors use, which enables those applications to run in a DB2 pureScale environment with minimal or no changes.
- Continuous availability
- The DB2 pureScale Feature provides a fully active-active configuration such that if one member goes down, processing continues at the remaining active members. During a failure, only data being modified on the failing member is temporarily unavailable until database recovery completes for that set of data, which is very quick. This is an advantage over some competing solutions in which an entire system freeze occurs as part of the database recovery process.
- Reduced total cost of ownership
- The DB2 pureScale interfaces easily handle the deployment and maintenance of components integrated within the DB2 pureScale Feature. This helps reduce what might amount to steep learning curves associated with deploying and maintaining with some competing technologies.
To understand these benefits, it helps to understand the basic architecture. Clients connect to a single database view comprised of four cluster members with shared storage. The clients connect anywhere. The DB2 member runs on several host computers with integrated cluster services. The members are connected over low-latency, high-speed connections to the PowerHA pureScale server. The logs and data-sharing architecture are centralized for all the clients. Figure 1 depicts the components that make up the DB2 pureScale configuration. Even though there are several advanced components, a significant portion of this architecture is transparent to the end user, because the DB2 pureScale Feature internally deploys and manages these components.
Figure 1. DB2 pureScale Feature topology overview
In Figure 1, clients can connect to any member. The DB2 pureScale Feature can automatically load balance the clients across the different members based on the utilization of each computer. If any host in the configuration fails, the DB2 pureScale Feature redirects clients among the active members on the remaining hosts.
Each DB2 member represents a DB2 processing engine. Up to 128 members can be deployed in a single DB2 pureScale configuration. The members cooperate with each other and with the PowerHA pureScale server to provide coherent access to the database from any member. Members can be added and removed as processing demands change. Modifications to the number of active members can be done without any impact to the clients.
Integrated with the DB2 pureScale Feature is a cluster services layer that provides failure detection, recovery automation, and a clustered file system. These technologies are integrated within the DB2 pureScale Feature, and they leverage IBM technologies optimized for DB2 software. The technologies include IBM Tivoli® Systems Automation for Multiplatforms (Tivoli SA MP), Reliable Scalable Cluster Technology (RSCT), and General Parallel File System (GPFS™). The DB2 pureScale Feature automatically deploys and configures these technologies according to a best-practice, pre-defined configuration. A customer does not need to determine how to configure the clustering technology that comes with the DB2 pureScale Feature, because the end user doesn't see it.
In the DB2 pureScale configuration, there is a need for communication between the members and the PowerHA pureScale servers. To make this communication as efficient as possible, the DB2 pureScale Feature leverages the RDMA technology. RDMA allows one computer to read or write to the memory of another computer without requiring any processor cycles on the target machine. This mechanism, in conjunction with extremely high-speed networks such as InfiniBand, allows for an extremely efficient transport layer, which enables the DB2 pureScale Feature to scale efficiently.
The PowerHA pureScale servers (also referred to as CF) provide a scalable and centralized locking mechanism to ensure data coherency. The servers also act as a fast cache for DB2 pages, leveraging RDMA technology to provide increased performance in situations where a physical disk operation might otherwise have been required. The PowerHA pureScale server and the efficient transport layer are two of the features that enable the DB2 pureScale Feature to scale so well, because each member does not have to negotiate with all other members when performing a task.
The DB2 pureScale Feature leverages a shared-disk technology. Any member can read or write to any portion of the database. If any member fails, the full set of data is still accessible from the other active members.
Deploying the DB2 pureScale Feature
Figure 1 shows four members for the sake of illustration, but the example deployment scenario described in this article deploys a two-member configuration.
This scenario deploys the DB2 pureScale Feature on a pair of P6 550 machines with each physical computer having two Logical Partitions (LPARs). Each LPAR has the following characteristics:
- It is on a public network that allows for client connectivity
- It is on an InfiniBand network to allow for high-speed, low-latency communication between the members and the PowerHA pureScale servers
- It has shared connectivity to a common set of disks
For the example scenario, ServerA and ServerB are connected to ServerC and ServerD by an InfiniBand network. The servers are connected to clients by the public network and by a storage area network.
Figure 2. Sample DB2 pureScale Feature hardware configuration
Table 1 shows the high-level configuration of each LPAR.
Table 1. Configuration overview
|OS level|| AIX® 6.1 TL3
| AIX 6.1 TL3
| AIX 6.1 TL3
| AIX 6.1 TL3
|Server type||Member 0||Primary CF||Member 1||Secondary CF|
|RAM||20 GB||12 GB||20 GB||12 GB|
|Shared disks|| hdisk2 -
64MB Disk leveraged by DB2 Cluster Services layer|
hdisk3 - 100 GB disk used to create a shared file system to hold shared instance files across hosts
hdisk4 - 200 GB disk used for DB2 data
hdisk5 – 20 GB disk used for DB2 transaction logs
Note: Disk sizes vary based on specific requirements
|Disk device driver||default MPIO (18.104.22.168)||default MPIO (22.214.171.124)||default MPIO (126.96.36.199)||default MPIO (188.8.131.52)|
For information about how to configure InfiniBand, refer to Appendix A.
Complete the following steps to prepare to install the DB2 pureScale Feature. Note that all commands listed during the installation steps should be executed as a user with root privileges unless specified otherwise.
- Ensure password-less ssh is setup within all the physical computers participating in a DB2 pureScale cluster. You can validate the ssh configuration by issuing the # ssh <target machine> hostname command from each computer to each computer in the cluster, and ensure it returns the valid hostname without any prompting.
- Make sure you have at least 5 GB of free space in the /tmp file system of each computer.
- Identify the disks that will be used for the DB2 pureScale Feature, and ensure each is tagged with a physical volume identifier (PVID) by completing the following steps:
- Enter the lspv command to list all physical volumes available on a computer, as shown in Listing 1.
Listing 1. List of physical volumes
# lspv hdisk0 00cc14e2ad7c570e rootvg active hdisk1 00cc14e2ba1a1dcd homevg active hdisk2 None None hdisk3 None None hdisk4 None None hdisk5 None None
hdisk2 to hdisk5 do not have a PVID identified in the second column.
- To add a PVID, run the chdev command from one computer in the cluster. The results are shown in Listing 2.
Listing 2. List from the chdev command
# chdev -l hdisk2 -a pv=yes # chdev -l hdisk3 -a pv=yes # chdev -l hdisk4 -a pv=yes # chdev -l hdisk5 -a pv=yes
- On each of the other computers in the cluster, run the commands in Listing 3 to remove the disk names.
Listing 3. Remove disk names
# rmdev -dl hdisk2 # rmdev -dl hdisk3 # rmdev -dl hdisk4 # rmdev -dl hdisk5
- Run the cfgmgr command from each computer in the cluster.
- Run the lspv command to display each hdisk's PVID that can uniquely identify the disk from any computer within the cluster, as shown in Listing 4.
Listing 4. Display PVIDs
# lspv hdisk0 00cc14e2ad7c570e rootvg active hdisk1 00cc14e2ba1a1dcd homevg active hdisk2 00cc14e23bb31057 None hdisk3 00cc14e23bb3b9ee None hdisk4 00cc14e23bb3e9dd None hdisk5 00cc14e23bb3e9df None
Tip: To validate the size of a disk, run the bootinfo -s hdisk2 command.
DB2 pureScale Feature installation steps
- Identify one of the LPARs as the installation-initiating host (IIH). This host runs the installation program for the DB2 pureScale Feature. The example scenario uses ServerA as the IIH.
- Ensure that the display is set to display graphical tools.
- Navigate to the ese_dsf folder on the downloaded product image or to the root directory of the product installation DVD, and run the /db2setup -t /tmp/db2setup.trc -l /tmp/db2setup.log command, which launches the DB2 graphical installer.
- The execution of db2setup takes you to a welcome screen, and an information panel is displayed as shown in Figure 3.
Figure 3. DB2 setup launchpad
- Click the various topics on the left pane to see more information. The Architecture Overview topic provides a high-level overview of the DB2 pureScale instance environment.
- Click Install a Product > Install New to install the DB2 pureScale Feature, as shown in Figure 4.
Figure 4. DB2 setup launchpad, selecting to install
A progress indicator bar appears at the bottom of the window. The setup wizard appears and provides a set of intuitive instructions to navigate through the remainder of the deployment.
- Click Next to continue the installation.
- Optionally, click View Features to see which features will be installed, as shown in Figure 5.
Figure 5. Setup wizard with View Features button
The next window displays the software licensing associated with the DB2 pureScale Feature, as shown in Figure 6.
Figure 6. Licensing terms
- Read the license, click the Accept radio button if you accept the licensing terms, and click Next.
- Select an installation method. For the example scenario, install the DB2 pureScale Feature directly from the graphical interface by selecting the first option, Install DB2 Enterprise Server Edition with the pureScale Feature, as shown in Figure 7.
Figure 7. Selecting the installation
- Click Next.
- Provide information for the installation directory of the DB2 binaries, and click Next. In most cases, the default path, /opt/IBM/db2/V9.8, is appropriate, as shown in Figure 8.
Figure 8. Select path
- Click Create a DB2 instance, and click Next, as shown in Figure 9.
Figure 9. Select Create a DB2 instance
- Enter the information of the instance owner, and click Next, as shown in Figure 10. Note that if an existing user is selected for the DB2 instance owner or for the DB2 fenced user, the user must exist on all the hosts with the same UID, GID, group name, and $HOME path. The hosts should not share the instance owner's $HOME directory, because that should be local to each host. If you want to create a new user, the defined new user must not exist in any of the hosts.
Figure 10. Set user information for the DB2 instance owner
- Enter the fenced user information, and click Next, as shown in Figure 11.
Figure 11. Set user information for the fenced user
The next steps use one of the predefined disks to create a shared
file system the DB2 pureScale instance environment uses for
instance files that are shared across all computers. The file system
will be mounted as
/db2sd<timestamp>, and the directory
/db2sd<timestamp>/<instance_name> will be the default
database path (as defined by the DFTDBPATH configuration parameter).
Another small disk will be leveraged for automatic internal cluster
- Provide the full path to the disks, and click Next, as shown in Figure 12.
Figure 12. Set up a DB2 cluster file system
- Add the remaining hosts that should be part of the DB2 pureScale cluster. By default, the IIH will already be included. For each host that you need to add, click Add, as shown in Figure 13.
Figure 13. Host list
- Enter the host name, and click OK.
- Repeat Steps 15 and 16 for each host that you need to add, as shown in Figure 14.
Figure 14. Adding remote hosts
Note that after each host is entered, the DB2 installation program does validation checking to ensure everything is configured properly for the new host, as shown in Figure 15.
Figure 15. Validation of installation settings
After adding all the hosts, the host list window appears with a check mark beside each host to confirm it has been validated, as shown in Figure 16.
Figure 16. Completed host list
- Check the configuration identified in the bottom portion of the window, and click Next. The DB2 installation program does one more validation of the passwordless ssh configuration. If there is more than one physical computer in the configuration, the installation wizard automatically assigns each PowerHA pureScale server to a different computer.
- Confirm the assignments in the Instance Settings section of the window.
- Optionally change which hosts serve which function in terms of members or PowerHA pureScale servers by clicking Advanced. It is strongly recommended that you have two PowerHA pureScale servers identified on two distinct physical computers to ensure that you do not have a single point of failure.
- Check and correct the summary of the values, and click Finish to start the installation, as shown in Figure 17.
Figure 17. Start copying files
During the actual installation process, a progress monitor bar enables you to see the progress of the installation, as shown in Figure 18. The installation takes several minutes as the DB2 pureScale Feature is being deployed to all the hosts in the configuration and as all the components are being deployed.
Figure 18. Installing task bar
- Upon successful installation of the DB2 pureScale Feature, click Finish. The installation of the DB2 pureScale Feature across all four hosts is complete with a DB2 instance ready for use, as shown in Figure 19.
Figure 19. Setup has completed successfully
DB2 pureScale Feature post-installation steps
With a DB2 pureScale Feature for Enterprise Server Edition instance ready for use, your next step is to create a file system to use for the data and the logs. You can create the file system using the db2cluster command. Complete the following steps from the root user.
- Create one file system for data and one file system for logs using the commands shown in Listing 5.
Listing 5. File system commands
#<DB2 Install Path>/bin/db2cluster -cfs -create -filesystem data -disk /dev/hdisk4 #<DB2 Install Path>/bin/db2cluster -cfs -create -filesystem log -disk /dev/hdisk5
The DB2 Installation Path in this example deployment is
/opt/IBM/db2/V9.8. The data and log file systems
are created by default under
/db2fs. They are accessible on all hosts in the DB2
- Modify the owner of the filesystem to be the DB2 instance owner so the DB2 instance owner has full access to this file system. For the example, db2sdin1 is the instance owner's name, and db2iadm1 is the instance owner's group name, as shown in Listing 6.
Listing 6. Assign filesystem owner
#chown db2sdin1:db2iadm1 /db2fs/data #chown db2sdin1:db2iadm1 /db2fs/log
- Start the DB2 instance by issuing the db2start command, as shown in Listing 7.
Listing 7. db2start command
> db2start 04/19/2010 11:02:08 0 0 SQL1063N DB2START processing was successful. 04/19/2010 11:02:08 1 0 SQL1063N DB2START processing was successful. SQL1063N DB2START processing was successful.
- Check the state of the DB2 pureScale instance at any point using the db2instance command. Tip: You can view the state of a DB2 pureScale cluster using the db2instance -list command.
- Create the database and move the logs to the log file system using the commands in Listing 8. Note that you must run the commands from member hosts, not CF hosts.
Listing 8. Move logs to log file system
> db2 create db testdb on /db2fs/data > db2 update db cfg for testdb using newlogpath /db2fs/log
- Catalog the client connections to any active pureScale members, and connect to the PowerHA pureScale server.
Using the DB2 pureScale Feature
The ease of installing and deploying the DB2 pureScale Feature reduces your total cost of ownership. This section offers more details about use-cases that demonstrate the added value of installing the DB2 pureScale Feature.
Adding and removing members
The DB2 pureScale Feature enables you to add members to the configuration quickly and without any data redistribution requirements. The DB2 installation binaries are automatically stored on the IIH, so they don't require access to the original installation media when you are adding members. Add a member by running the command in Listing 9 from the IIH.
Listing 9. Adding a member
db2iupdt -d -add -m ServerX:ServerX-ib0 db2sdin1
Members can be started or quiesced transparently to the application such that the application is unaware a change has occurred.
Automatic workload balancing
With the DB2 pureScale Feature, you can dynamically distribute a workload across all the active members based on the computer utilization characteristics of the different machines. Multi-threaded CLI applications by default have connection-level workload balancing without any changes. This workload balancing can be modified such that it applies at the transaction level instead of the connection level. For multi-threaded Java® applications, set enableSysplexWLB=true in the connection string to take advantage of transaction-level workload balancing.
As additional members are started, clients automatically route to the new member without any interruption of service. Also, members can be stopped without the application knowing.
You can also configure clients to have a preference to which members they should connect to. This feature is referred to as client affinity, and it can be beneficial if a partitioned workload already exists.
Note: To take advantage of DB2 pureScale Feature benefits such as transaction-level workload balancing or client affinity, the minimum client level should be Version 9.7 Fix Pack 1 or the correlating JCC level. To correlate the JCC levels included in the various fix pack levels, see Resources.
At times, it is critical to apply maintenance to a system without negatively impacting the client applications. Stealth maintenance basically allows all transactions on a member to complete and then transparently routes that application to another member. To drain member 1 in the example scenario, you can run the db2stop member 1 quiesce command.
Sometimes a unit of work (UOW) is started for a user session, but the UOW is not committed or rolled back. The db2stop quiesce command waits for that UOW to be completed before stopping that member. You can specify a timeout value such as ten minutes that permits the application ten minutes to complete the UOW and, if after ten minutes the UOW is not completed, the DB2 software automatically forces off that application. Any applications that complete the UOW within the ten minutes are automatically rerouted to the active members. For example, to drain member 1 with a ten minute timeout, run the command db2stop member 1 quiesce 10.
One of the significant value propositions of the DB2 pureScale Feature is the high-availability characteristics integrated into the architecture. The DB2 pureScale cluster services automatically monitor all necessary resources and restart them as needed. Applications that are connected to a failing member are rerouted to an active member where the application can reissue any failed transactions. Applications connected to a non-failing component are not impacted.
One of the distinguishing factors of the DB2 pureScale Feature compared to most competing technologies is that upon a member failure, no cluster-wide freeze occurs. Only data in the process of being updated on the failing member is temporarily unavailable until recovery is complete. Applications on active members trying to access the locked data on the failing member are briefly in a lock-wait state, and by default they do not receive any errors. The recovery is completed quickly so that data availability through a member failure looks similar to the hypothetical representation in Figure 20.
Figure 20. Typical data availability pattern during member recovery
Many customers require a disaster recovery solution to meet their business continuity requirements. The DB2 pureScale Feature leverages remote disk-mirroring technology and is also designed to work with database replication products. If the entire primary site that is running a DB2 pureScale instance fails, you can leverage the remote site to allow business operations to continue, as shown in Figure 21. For the example scenario, the DB2 pureScale production instance is on site A, which feeds into the storage. With disk mirroring, the DB2 pureScale DR instance site B also feeds into storage. The DB2 pureScale Feature can also leverage the traditional database backup, restore, and roll-forward functions that also enable disaster recovery solutions.
Figure 21. Typical disaster recovery setup
The DB2 pureScale Feature for Enterprise Server Edition provides a database solution that meets the needs of the most demanding customers. This article described some of the key benefits of using DB2 pureScale Feature. It leverages the PowerHA pureScale server and RDMA technologies, allowing it to scale effectively to meet the growing and dynamic needs of different organizations. You can add members to the DB2 pureScale environment without any impact to existing applications to meet the demands of peak processing times. The DB2 pureScale Feature automatically balances the workload across all DB2 members in the cluster without any changes on the application side, taking full advantage of the additional processing capacity. If a DB2 member fails, applications are automatically routed among the other active members. When the failed member host comes back online, applications are transparently routed to the restarted member.
Appendix A. Configuring and deploying InfiniBand
The DB2 pureScale Feature leverages an InfiniBand network to allow for optimal communication between members and PowerHA pureScale servers. This appendix describes the high level steps of deploying InfiniBand.
- Deploy the InfiniBand switch with an active subnet manager from the hardware management console, as shown in Listing 10. You need HMC Version 7, Release 3.5 or later. In the example scenario, a single InfiniBand card is shared across two LPARs. This step enables the InfiniBand resources to be shared by either 25%, 50%, or 75% of the InfiniBand card.
Listing 10. Deploying the InfiniBand switch
chsyscfg -m machine1-8204-E8A-SN0657742 -r -sys -i "hca_bandwidth_capabilities=25/50/75" chsyscfg -m machine2-8204-E8A-SN0657742 -r -sys -i "hca_bandwidth_capabilities=25/50/75"
In this example, ServerA and ServerB reside on machine1 and ServerC and ServerD reside on machine2. You can instead specify values other than 25%, 50%, or 75%.
- Repeat Step 1 for each physical computer.
- Complete the following steps from the hardware management console's
graphical interface to allocate
the InfiniBand resources for each LPAR so that each LPAR is assigned 50%.
- Select the checkbox beside the LPAR
- Click Configuration > Manage Profiles.
- Select the default profile.
- Click HCA > Configure.
- Assign each LPAR a unique GUID index within each physical computer to distinguish the different LPARs using InfiniBand.
- Select Medium (50%) capability as the allocation of the InfiniBand resources for each LPAR, which is the middle setting from Step 1 .
- Repeat Steps a-f for each LPAR.
- Shut down and reactivate each of the LPARs from the HMC.
- Download the required uDAPL and InfiniBand packages as documented in the DB2 pureScale Feature documentation (see Resources), install these on all LPARs, and reboot each LPAR.
- Enable InfiniBand on each LPAR by running the smitty icm command and selecting Add an InfiniBand Communication Manager.
- Complete the following steps to set up the InfiniBand network
interface on each LPAR.
- Run the smitty inet command.
- Click Change/Modify Characteristics.
- Select ib0.
- Reboot the LPAR. Note that you must change the Internet address and netmask. For the example scenario, assign 10.1.1.7 on serverA, 10.1.1.8 on serverB, 10.1.1.9 on serverC, and 10.1.1.10 on serverD for the Internet addresses, assign 255.255.255.0 for the netmask on each, and use the defaults for all other settings.
- Repeat Steps a-d for each LPAR.
- Ensure the /etc/dat.conf file has a format similar to Listing 11 for each LPAR. For the example scenario, make the host channel adapter /dev/iba0 , on port 1 using interface ib0.
Listing 11. Settings in /etc/dat.conf file
hca0 u1.2 nonthreadsafe default /usr/lib/libdapl/libdapl.a(shr_64.o) IBM.1.1 "/dev/iba0 1 ib0" " "
- Edit the /etc/hosts file on each LPAR for routing purposes by following the example in Listing 12.
Listing 12. Editing the /etc/hosts file
10.1.1.1 servera-ib0.<domain_name> servera-ib0 10.1.1.2 serverb-ib0.<domain_name> serverb-ib0 10.1.1.3 serverc-ib0.<domain_name> serverc-ib0 10.1.1.4 serverd-ib0.<domain_name> serverd-ib0
- Optionally validate the state of the InfiniBand on each LPAR by running the ibstat -v command from a user with root privileges. Verify that port 1 is active and that the link is up. You can also perform a ping test using the addresses and hostnames defined in the /etc/hosts file.
Appendix B. Supported AIX server stacks
The DB2 pureScale Feature currently supports the following POWER6® models:
The DB2 pureScale Feature currently supports the following POWER7™ models.
Appendix C. Supported storage subsystems
The DB2 pureScale Feature supports a wide range of IBM storage, including the IBM DS3000, DS5000, DS8000®, and XIV® lines of storage, as well as storage provided by others vendors, including EMC, Hitachi, and others. To leverage the DB2 pureScale Feature with rapid fencing that is required for ultra-fast failover times, the disk storage should be validated with SCSI-3 persistent reserve with an underlying clustered file system. See Resources for the storage subsystems validated with SCSI-3 persistent reserve.
- Refer to the GPFS FAQs for the storage subsystems validated with SCSI-3 persistent reserve.
- Query the DB2 JDBC/JCC Driver Versions to correlate the JCC levels included in the various fix pack levels.
- Learn more about Information Management at the developerWorks Information Management zone. Find technical documentation, how-to articles, education, downloads, product information, and more.
- Stay current with developerWorks technical events and webcasts.
Get products and technologies
- Download Download a free trial version of DB2 9.7 for Linux, UNIX, and Windows
- DB2 Express-C 9.7 Download DB2 Express-C 9.7, a no-charge version of DB2 Express database server for the community.
- Participate in the discussion forum.
- Check out the developerWorks blogs and get involved in the developerWorks community.
Dig deeper into Information management on developerWorks
Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.
Keep up with the best and latest technical info to help you tackle your development challenges.
Software development in the cloud. Register today to create a project.
Evaluate IBM software and solutions, and transform challenges into opportunities.