The IBM DB2 pureScale Feature

Designed for organizations that run online transaction processing (OLTP) applications, IBM DB2 pureScale feature offers clustering technology that helps deliver high availability and scalability, which is transparent to applications. DB2 pureScale Feature leverages the industry standard set by IBM DB2 for z/OS and brings best-of-breed architecture to the distributed platform. Currently, DB2 pureScale Feature is available as an option on IBM DB2 9.7 Enterprise Server Edition, Advanced Enterprise Server Edition, and Workgroup Edition. DB2 pureScale Feature lets you scale out your database, allowing OLTP applications to take advantage of the added capacity — without major changes or tweaking. You simply add more computing resources to the cluster when you need to meet increased workload volumes. With the DB2 pureScale Feature, your applications can access data even when one or more members in the cluster experience an outage. Transparent workload balancing allows the applications to function during the outage, and when failed members are brought back online, they automatically begin processing incoming database requests.

For a typical OLTP-type workload running with DB2 pureScale Feature as shown in Figure 1, DB2 client requests are sent to DB2 members through the local area network (Ethernet). The Remote Direct Memory Access (RDMA) high-speed interconnect fabric is then used between DB2 members and the cluster caching facility (CF) for synchronizing the locking and caching information across the cluster. CF is used to synchronize locking through the global lock manager (GLM) to coordinate access to the same rows by different members. CF also provides a group buffer pool (GBP) that members can use to emulate cluster-wide shared memory and keep pages consistent across all members. A DB2 member uses cluster services (CS), which provide heartbeat-based failure detection to automatically initiate the necessary recovery operations. DB2 members read and write to the same database image on storage (Fibre Channel–based SAN) that is shared among all database nodes in cluster. See DB2 pureScale overview for details.

Back to top

IBM BladeCenter

Unlike a stand-alone server that needs multiple power supplies, fans, individual systems management, numerous cables, and a lot of space, IBM BladeCenter is compact and simple. BladeCenter integrates servers, storage, and networking to help you reduce complexity, simplify IT management and reduce costs, through a flexible and highly scalable design. With an intelligent system design, IBM BladeCenter includes open connectivity and multiple layers of redundancy and reliability combined with advanced management tools. The HS22 blade server, discussed in this article, supports up to two high-performance Intel® Xeon® processors. It offers outstanding performance in an energy-efficient design for database and other enterprise applications.

Back to top

IBM DB2 pureScale on IBM BladeCenter

Implementing IBM DB2 pureScale Feature on IBM BladeCenter can help reduce the risk and cost of database growth due to increased workload demands, while providing desired availability and application transparency. As workload demands increase, you need an efficient approach to scale capacity while minimizing hardware integration complexity. With IBM BladeCenter you can handle increasing workload demands by simply adding blade servers to your existing DB2 pureScale cluster with few or no changes to applications. Together, DB2 pureScale Feature with its efficient scale-out architecture and IBM BladeCenter with its growth simplicity, provide an optimal environment for an OLTP workload.

Back to top

Reference architecture

This reference architecture takes you through seven steps required to deploy IBM DB2 pureScale Feature on IBM BladeCenter. The reference architecture covers a set of solution templates, the hardware and software components, and required hardware integration steps. To cope with workload-demand increase, this article outlines how to achieve elasticity after the initial implementation.

Follow these steps to set up your production-ready environment with DB2 pureScale on IBM BladeCenter:

1. Select a solution template matching your preferred configuration.
2. Decide on a cluster high-speed interconnect fabric.
3. Select the type of Fibre Channel storage.
4. Acquire the hardware components for the selected solution template.
5. Integrate and configure IBM BladeCenter.
6. Install a supported Linux® distribution and other prerequisites.
7. Install the DB2 pureScale Feature and configure a production-ready instance.

Step 1: Select a solution template matching your preferred configuration

The architecture of DB2 pureScale Feature allows you to create a cluster that matches your workload demands. This section contains several solution templates that provide an easy overview of individual components that are needed for configuring DB2 pureScale Feature on IBM BladeCenter.

The two-two solution template shown in Figure 2 is the minimum configuration consisting of two blade servers, where each blade server would act both as a DB2 member and a CF. Even with the loss of one blade server in this configuration, the DB2 instance would continue to be available.

The three-five solution template shown in Figure 3 demonstrates continuous high availability during hardware failure or hardware maintenance of one blade server. This configuration uses a total of five blade servers, where three are designated as members and two as CFs.

The four-six solution template shown in Figure 4 represents a four-member and two-CF configuration. The fourth member is added by executing a simple DB2 command. See the Elastic size clusters section for more details.

The eight-ten solution template shown in Figure 5 consists of eight members and two CFs.

Table 1 shows the number of DB2 members, cluster caching facilities, blade servers, and types of DB2 licenses needed for the solution templates described above.

Step 2: Decide on a cluster high-speed interconnect fabric

The performance of a DB2 pureScale cluster is closely tied to the latency of the interconnects between the DB2 members and the CF. IBM DB2 pureScale Feature uses the User Direct Access Programming Library (uDAPL) to exploit RDMA technology, which enables members to coordinate page caches and locks with the CF. Use of RDMA helps dramatically reduce the communications overhead required to request and transfer data. The DB2 pureScale Feature supports two such RDMA-capable networking transport fabrics: InfiniBand, and 10 Gb RDMA over Converged Ethernet (RoCE). Both are excellent choices when it comes to a low-latency fabric. However you must ensure that the protocol of the adapters in the individual blade servers matches the switch modules in the chassis. For example, if InfiniBand is selected, then both the expansion cards and switch modules must be InfiniBand. The cluster interconnect must be an InfiniBand network or a 10 Gigabit Ethernet (10GbE) network. A mixture of these two networks is not supported. See the requirements for the cluster interconnect network in the DB2 pureScale Feauture Information Center.

Step 3: Select the type of Fibre Channel storage

DB2 pureScale Feature uses a shared-disk architecture, which allows direct access to all data by all servers in the cluster. All of the solution templates include Fibre Channel switch modules and Fibre Channel adapters as shown in Table 5. The Fibre Channel switches allow the blade servers to connect to Fibre Channel storage controllers. See Shared Storage Considerations for DB2 pureScale in the Information Center for a list of supported storage subsystems. This reference architecture was validated with the IBM System Storage DS3500 due to its price-to-performance ratios, functionality, scalability, and, most importantly, its support of SCSI-3PR. The DS3500 storage controllers and multipath I/O driver combination support specialized SCSI commands, known as SCSI-3 Persistent Reserve (PR). The SCSI-3 PR enables DB2 cluster services to perform fast I/O fencing in as little as one-to-two seconds. That means that if there is a failed member in the DB2 pureScale instance, DB2 cluster services makes sure that the failed member cannot modify shared data on disk.

Step 4: Acquire the hardware components for the selected solution template

The solution templates described in Table 1 require various hardware components. The quantities and types of these components are shown in Table 2.

By following the previous steps, you can identify and determine your desired solution template, cluster high-speed interconnect, and type of the storage subsystem. To identify individual type and quantity of hardware types to validate this reference architecture, use the following tables.

Table 3 lists the part numbers for IBM BladeCenter chassis, blade servers, Ethernet switch module for corporate connectivity, power distribution units and SFP transceiver.

Table 4 provides the part numbers for cluster high-speed interconnect fabric. The Infiniband is used for this reference architecture.

Table 5 provides the part numbers for Fibre Channel switch module and adapter.

Once all the required hardware types are identified, you need to gather specifications and lists of options for all these components.

To confirm the quantity, type and specifications for IBM BladeCenter hardware components, see the IBM Redbooks® publication IBM BladeCenter and System x Reference Documents.

For interoperability information see the BladeCenter Interoperability Guide.

Refer to Appendix A for detailed information on various hardware types.

Step 5: Integrate and configure IBM BladeCenter

To install and configure IBM BladeCenter H components, perform the following integration steps:

1. Install the BladeCenter H chassis. To install and configure BladeCenter H, see IBM BladeCenter H Type 8852 and Installation and User's Guide- IBM BladeCenter H.
2. Install the adapter cards in the blade servers. Before inserting the blade servers into the BladeCenter H chassis, make sure to install the required adapter cards, such as InfiniBand or 10 Gb Ethernet (RoCE) and Fibre Channel cards in the blade servers. These cards have their own connectors to BladeCenter’s midplane and a PCI Express socket on the blade server. Depending on choice of cluster high-speed network fabric, install any of the following adapter cards:
   • Cluster high speed interconnect
     • The 2-Port 40 Gb InfiniBand Expansion Card (CFFh) for IBM BladeCenter with a dual-port InfiniBand Host Channel Adapter (HCA), combined with the QDR switch, delivers end-to-end 40 Gb bandwidth per port. The adapter uses the CFFh form factor. Read the 2-Port 40 Gb InfiniBand Expansion Card for IBM BladeCenter for specifications and see Installing a Horizontal Compact-Form-Factor Expansion Card for installation of this adapter card.
     • The Mellanox 2-Port 10 Gb Ethernet Expansion Card for IBM BladeCenter is based on ConnectX-2 EN technology with RoCE specification. This adapter card provides efficient data transfer with very low latency. Blade servers in the BladeCenter H chassis can fully utilize both 10 Gbps ports to achieve up to 40 Gbps duplex bandwidth. See the specification of Mellanox 2-port 10Gb Ethernet Expansion Card for IBM BladeCenter and follow the Installation Guide to install this adapter card.
   • Fibre Channel
     • The QLogic 8 Gb Fibre Channel Expansion Card (CIOv) provides high-speed access for IBM blade servers to connect to Fibre Channel storage area network (SAN). In this reference, this card is installed in the PCI Express CIOv slot of the HS22 blade server, which provides connections to switch modules in bays 3 and 4 of BladeCenter H. See the specification of QLogic 8Gb Fibre Channel Expansion Card (CIOv) for IBM BladeCenter and refer to Installing a CIOv Form-Factor Expansion Card to install this adapter card.

   Once all the adapter cards are installed, close the blade server cover, as described in the Information Center.

3. Install the blade servers. With all required adapter cards installed in the blade servers, insert the blade servers into the chassis. See Installing the blade server in a BladeCenter Unit and consult the HS22 Blades section in Appendix A for proper handling and installation of the blade servers.
4. Install the switch modules. The IBM BladeCenter H chassis has a total of 10 I/O bays and each blade server bay has a total of eight dedicated connection paths to the I/O modules. See Installing an I/O module in BladeCenter H– page 27 (Adobe page 43) within the “BladeCenter Type 8852 Installation User's Guide”.

   To install switch modules, the following I/O bays are used:

   • Bay 1 and 2

     Bay 1 and 2 contain the standard Ethernet-compatible I/O module that routes internally to onboard Ethernet controllers on the blade servers. The BNT Layer 2/3 GB Copper Ethernet Switch Module is used for corporate network. To install Ethernet switch module, see Appendix A. Note that external link is required for Ethernet switch to connect to the corporate network.

   • Management module bay 1 and 2 (MM1,MM2)

     The management module bays MM1 and MM2 are used for IBM BladeCenter H Advanced Management Module (AMM). The AMM is a BladeCenter module that is used to configure and manage all BladeCenter components. The connection to the management network is also established through this module. To install the AMM, see AMM install guide for BladeCenter.

   • Bay 3 and 4

     Bay 3 and 4 are used for Fibre Channel connectivity, which is routed internally to the PCI-X connector on the blade servers. The QLogic 20-port 4/8 Gb SAN Switch Module has 14 internal ports that operate as fabric ports at 2 Gbps, 4 Gbps or 8 Gbps, depending on which Fibre Channel expansion cards are installed, and the location of the I/O module bays in which the switch is installed. It also has six external ports that operate at a maximum speed of 8 Gbps. To install the Fabric Channel switch module see Appendix A.

   • Bay 7 through 10

     I/O bays 7 through 10 are used for high-speed switch modules and depending on your selection of the high speed interconnects fabric, the Infiniband switch or 10 Ge Ethernet switch can be installed. In this reference architecture the Voltaire 40 Gb InfiniBand switch module for BladeCenter is used.
     

     IBM BladeCenter H can house 14 blade servers and each blade server has an InfiniBand adapter card. Each card has two InfiniBand ports that connect to the two Voltaire 40 Gbps InfiniBand switch modules in bay 7-8 at the top and bay 9-10 at the bottom. Port 1 of each card connects to the switch module installed in bay 7-8 at the top, and Port 2 connects to the switch module installed in bay 9-10 at the bottom. To install the InfinBand switch module see Appendix A.

5. Provide external links for SAN
   

   With Fibre Channel expansion cards and switch modules all installed, the external link to the external storage needs to be established. It is recommended to use two Fibre Channel switch modules for redundancy and load-balancing purposes. In this reference, the QLogic switch modules in IBM BladeCenter H connect to the DS3500 disk storage. This disk storage has two controllers that are used to connect to the two Fibre Channel switches. See QLogic 20–port 4 Gb SAN Switch Module for IBM BladeCenter and read section 4.8 of the IBM System Storage DS3500 Introduction and Implementation Guide.

6. Set up the Advanced Management Module and power up BladeCenter.
   

   The BladeCenter components are configured and managed through the BladeCenter Advanced Management Module (AMM). To configure the AMM, refer to the BladeCenter Advanced Management Module User's Guide.

   As an example, Figure 6 shows the status and location of the I/O bays through the GUI interface of the AMM.

   
   
   Figure 6. Figure 6. I/O module locations within BladeCenter H
   
   
   

   At this point, the BladeCenter chassis installation and connection to an external storage are all complete. See Chapter 4 of Installation and User's Guide before proceeding to power on the BladeCenter.

   For a detailed description of the I/O modules and adapter cards described, refer to IBM BladeCenter H Chassis and IBM BladeCenter HS22 blade servers.

7. Configure the SAN switch
   

   The Fibre Channel switch modules need to be aware of the shared disk storage. SAN zoning is recommended to configure the logical unit number (LUN) from the shared disk storage so that all the blade servers can see the shared disks. You will need to implement a SAN zoning configuration on the QLogic Fibre Channel switch that is connected to the storage subsystem such as DS3500. For further details on SAN zoning and general guidelines for implementing a zoning configuration, refer to section 4.8 of the IBM System Storage DS3500 Introduction and Implementation Guide. Also the RAID configuration is recommended to provide maximum redundancy and availability, such as RAID-6.

   To configure the QLogic Switch refer to QLogic 20–port 4 Gb SAN Switch Module for IBM BladeCenter.

Step 6: Install a supported Linux distribution and other prerequisites

Installation of the DB2 pureScale Feature requires that your system meet certain network, hardware, firmware, storage, and software requirements. It is important to set up and configure the required prerequisites as outlined in the Information Center section Pre-installation Checklist for DB2 pureScale Feature (Linux).

Step 7: Install the DB2 pureScale Feature and configure a production-ready instance

After completing steps one through six above, you can install the DB2 pureScale Feature. To install the DB2 pureScale Feature be sure to follow the Information Center section Planning Consideration for IBM DB2 pureScale Feature and necessary Preinstallation Tasks Before Installing DB2 pureScale Feature.

Once the planning and preinstall tasks have been completed, refer to Installing or Upgrading to the DB2 pureScale Feature to install DB2 pureScale Feature. Note that when you install the DB2 pureScale Feature from one host, the installation process also installs all of the components of DB2 pureScale Feature across all hosts that you specify as part of the DB2 pureScale environment. To see an example of installation on Linux, refer to Deploy the DB2 pureScale Feature on Linux.

A successful installation of the DB2 pureScale Feature using either the DB2 setup wizard or a response file creates a DB2 pureScale instance by default. However you will need to create an instance manually if you installed the DB2 pureScale Feature using the db2_install command. To manually create a DB2 pureScale instance, refer to the steps outlined in Creating a DB2 pureScale Instance.

Back to top

Elastic size clusters

Regardless of which solution template you choose, the reference architecture facilitates the simplified growth or shrinkage of capacity, referred to here as elastic size cluster. You can grow capacity using a DB2 command provided that the additional blade server has been provisioned in a similar manner to steps 5 and 6. To add DB2 members to an existing DB2 pureScale cluster quickly, see Adding a New DB2 Member for details.

You can shrink capacity for all solution templates except two-two by using a DB2 command to remove a member from the cluster. This also allows you to optionally remove the freed-up blade server from the chassis. To drop a DB2 member from an existing pureScale cluster, see Dropping a DB2 Member.

Back to top

Conclusion

In this reference architecture you learned that application transparency, scalability and continuous availability are among several other important elements for an OLTP-type workload environment which are addressed with DB2 pureScale Feature on IBM BladeCenter. Following certain integration steps, you can install, configure, and validate all the required hardware and software stacks to arrive at a production-ready environment with DB2 pureScale on IBM BladeCenter.

Back to top

Appendix A

IBM BladeCenter H chassis

• IBM BladeCenter H
• IBM BladeCenter H Installation and User's Guide
• IBM BladeCenter H Problem Determination and Service Guide

IBM BladeCenter HS22 blade servers

• IBM BladeCenter HS22 Models
• IBM BladeCenter HS22
• IBM BladeCenter HS22 Installation and User's Guide
• Problem Determination and Service Guide

High-speed switch modules

BladeCenter H high-speed switch modules require that the blade server have the matching high-speed adapter card. Mixing of different high-speed adapter cards in the same BladeCenter chassis is not supported.

InfiniBand switches

InfiniBand is used to communicate among DB2 members and CF. The connection of the InfiniBand switch with InfiniBand cards is all internal. A high-speed InfiniBand 4X QDR switch module from Voltaire allows full InfiniBand 4X QDR connectivity to each blade server. Externally the switch has 16 QSFP ports, all 4X QDR capable. See below for more information.

• Voltaire 40 Gb InfiniBand Switch Module for IBM BladeCenter
• Install and Enable the Voltaire 40 Gbps InfiniBand Switch Module

10Gb Switch Module

The BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter is the first 10 Gb switch for IBM BladeCenter that is convergence ready

• http://www.redbooks.ibm.com/abstracts/tips0708.html

SAN switches

• QLogic 20-Port 8 Gb and 4/8 Gb SAN Switch Modules for IBM BladeCenter
• IBM BladeCenter Open Fabric
• 20-port 8 Gb SAN Switch Module Installation and User's Guide

Ethernet switches

Dual integrated gigabit Ethernet controllers on each blade server connect to optional Gb Ethernet switch modules. The Ethernet adapter is connected to the corporate network, which DB2 clients can use to connect with the DB2 pureScale instance. Note that the DB2 pureScale Feature automatically routes connection requests to the member with the lowest workload or you can specifically connect to an active member in the DB2 pureScale instance.

• BNT Layer 2/3 Copper and Fiber Gigabit Ethernet Switch Modules for IBM BladeCenter
• BNT Layer 2/3 Copper Gb Ethernet Switch Module for IBM BladeCenter

Adapters

InfiniBand (CFFh)

ConnectX® 40Gbps InfiniBand dual-port adapters for IBM BladeCenter H deliver low-latency and high-bandwidth for performance-driven server. Clustered databases, parallelized applications and transactional services applications will achieve significant performance improvements resulting in reduced completion time and lower cost per operation.

• 2-Port 40 Gb InfiniBand Expansion Card (CFFh) Layout
• Dual-port 20 and 40 Gbps InfiniBand Mezzanine Adapters for IBM BladeCenter

Fibre Channel (CIOv)

The QLogic 8 Gb Fibre Channel Expansion Card (CIOv) for IBM BladeCenter enables high-speed access for IBM blade servers to connect to a Fibre Channel SAN. When compared to previous-generation 4 Gb adapters, the new adapter doubles the throughput speeds for Fibre Channel traffic.

QLogic 8 Gb Fibre Channel Expansion Card (CIOv) for IBM BladeCenter

Storage

Internal storage

Standard: SAS drives 146 GB SAS

External storage

DS3500 disk storage

• IBM System Storage DS3500

Note: When using SANs, multiple redundant data paths (multipaths) will exist between the switch module and storage subsystem such as DS3500. This is to help avoid interruptions in data flow should a hardware failure occur. For the Linux OS, the device mapper multipath (DM-MP) application is used for configuring the multipath I/O.

30A/250 V PDUs

To provide power to IBM BladeCenter, there are two power distribution units (PDUs).

Product: 30amp/250V Front-end PDU with NEMA L6-30P connector

Manufacturer: IBM

Part Number: 39Y8939

Refer to the guides below to plan power requirements for the required PDUs.

• IBM System x PDU Guide
• IBM System x and BladeCenter Power Configurator

Resources

Learn

• Check out the IBM BladeCenter Product Publications for quick reference guide on different parts of IBM BladeCenter.
  
  
• The Compatibility for IBM BladeCenter products guide provides you with the list of compatible product mode and types for IBM BladeCenter.
  
  
• Check out the IBM BladeCenter HS22 Technical Introduction to plan for installation and configuration of the IBM BladeCenter.
  
  
• See IBM BladeCenter H Problem Determination and Service Guide for troubleshooting the IBM BladeCenter H.
  
  
• Read the IBM BladeCenter H chassis documentation for installation and configuration of IBM BladeCenter H.
  
  
• Refer to the IBM BladeCenter Advanced Management Module documentation for complete installation, configuration and usage of IBM BladeCenter Advanced Management Module(AMM).
  
  
• Read the developerWorks article What is IBM DB2 pureScale for an overview of the IBM DB2 pureScale architecture.
  
  
• Get the information you need on Application cluster transparency with the IBM DB2 pureScale feature and related information.
  
  
• Read the developerWorks article Deploying the IBM DB2 pureScale Feature on Linux for an example of IBM DB2 pureScale deployment scenario on Linux platform.
  
  
• See the developerWorks article Solving Problems in the DB2 pureScale Cluster Services Environment for a widely used commands to troubleshoot problems within the IBM DB2 pureScale environment.
  
  
• See the developerWorks article Administer the IBM DB2 pureScale Feature Using Basic Commands for some examples of IBM DB2 pureScale administration activities using the available commands.
  
  
• Read the Deploying the DB2 pureScale Feature article for a sample deployment scenario based on P6 550s servers connected to a DS5300 storage controller.
  
  
• Get the detail you need on various licensing information for IBM products available for your reference.
  
  
• Learn more about Information Management at the developerWorks Information Management zone. Find technical documentation, how-to articles, education, downloads, product information, and more.
  
  
• Stay current with developerWorks technical events and webcasts.
  
  
• Follow developerWorks on Twitter.
  
  

Get products and technologies

• Build your next development project with IBM trial software, available for download directly from developerWorks.
  
  

Discuss

• Participate in the discussion forum.
  
  
• Check out the developerWorks blogs and get involved in the developerWorks community.
  
  

About the authors

Mel is a software engineer at the IBM Toronto Lab in Canada. He is part of the Systems Optimization Competency Center, specializing in workload- optimized solutions. Previously, he worked in software service group, specializing in DB2 and WebSphere Application Server performance.

Miso has been with IBM since 2000. Over the years he has worked on diverse set of projects, with focus on DB2 in the areas of performance, integration with and exploitation of hardware and design of workload optimized systems. Currently, Miso manages the DB2 Performance Benchmarking team.

Rate this article

Comments

Back to top

Table of contents

• The IBM DB2 pureScale Feature
• IBM BladeCenter
• IBM DB2 pureScale on IBM BladeCenter
• Reference architecture
• Elastic size clusters
• Conclusion
• Appendix A
• Resources
• About the authors
• Comments

Next steps from IBM

• Try: DB2 Enterprise
• Demo: Create an electronic form solution with DB2 pureXML and Lotus Forms
• Community: DB2
• Buy: DB2 Enterprise

Dig deeper into Information Management on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers