Setting up IBM Spectrum Scale (GPFS) for IBM Db2 Warehouse

If you want to deploy Db2® Warehouse in an MPP environment, you must set up a POSIX-compliant cluster file system. One of the storage technologies that you can use for this file system is IBM Spectrum Scale, formerly known as IBM® General Parallel File System (GPFS).

Before you begin

Set up one or more storage devices that support Spectrum Scale, such as IBM Cloud®m> block storage. For information about how to set up IBM Cloud block storage, see Setting up IBM Cloud block storage for IBM Db2 Warehouse.

Obtain the Spectrum Scale software and license.

About this task

In the following procedure, an example of a three-node system is used.

For more information about all of the mm* commands (such as mmcrcluster) that are used in the following procedure, see the IBM Spectrum Scale documentation.

Ensure that any files that you create in the following procedure are in the same working directory.

The following instructions apply to CentOS and RedHat operating systems. Instructions for other operating systems might be different.

Procedure

On each node, create an /etc/hosts file that has the host names and IP addresses of all the nodes.

An example follows:


[root@bluhelix16 ~]# cat /etc/hosts 
127.0.0.1 localhost.localdomain localhost 
10.90.61.40 bluhelix10 
10.90.61.49 bluhelix11 
10.90.61.44 bluhelix12

Configure passwordless SSH by performing the following substeps. The nodes in the cluster must be able to communicate with each other without the use of a password for the root user and without the remote shell displaying any extraneous output.
1. Generate an SSH key for the system by issuing the following command:
```
ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa -q -N "" 
```
2. Copy the public key of the SSH key to the ~/.ssh/authorized_keys directory by issuing the following command:
```
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
```
3. Change the permissions of the ~/.ssh/authorized_keys directory by issuing the following command:
```
chmod 600 ~/.ssh/authorized_keys
```
4. Copy the ~/.ssh directory to all nodes in the cluster.
5. Test the SSH setup to ensure that all nodes can communicate with all other nodes. Test by using short names, fully qualified names, and IP addresses. For example, assume that the environment has three nodes: node1.mydomain.com:10.0.0.1, node2.mydomain.com:10.0.0.2, and node3.mydomain.com:10.0.0.3. Repeat the following test by using the short names (node1, node2, and node 3), the fully qualified names (node1.mydomain.com, node2.mydomain.com, and node3.mydomain.com), and the IP addresses:
```
#!/bin/bash

# Edit node list
nodes="node1 node2 node3"

# Test ssh configuration
for i in $nodes
do for j in $nodes
 do echo -n "Testing ${i} to ${j}: "
 ssh ${i} "ssh ${j} date"
 done
done
```
  Sample output follows:
```
Testing node1 to node1 Wed Oct 15 10:14:34 CDT 2016
Testing node1 to node2 Wed Oct 15 10:14:34 CDT 2016
Testing node1 to node3 Wed Oct 15 10:14:35 CDT 2016
Testing node2 to node1 Wed Oct 15 10:14:35 CDT 2016
Testing node2 to node2 Wed Oct 15 10:14:36 CDT 2016
Testing node2 to node3 Wed Oct 15 10:14:36 CDT 2016
Testing node3 to node1 Wed Oct 15 10:14:37 CDT 2016
Testing node3 to node2 Wed Oct 15 10:14:37 CDT 2016
Testing node3 to node3 Wed Oct 15 10:14:38 CDT 2016
```

On all nodes, install the IBM Spectrum Scale binary file. To find the kernel version, you can use the uname -r command.


yum -y install unzip ksh perl libaio.x86_64 net-tools m4 kernel-devel gcc-c++ kernel-devel-kernel_version
rpm -ivh gpfs/*.rpm 
rpm -Uvh gpfsUpdates/*.rpm 
sed -i 's/)/) Red Hat Enterprise Linux/g' /etc/redhat-release 
/usr/lpp/mmfs/bin/mmbuildgpl

On the head node only, configure Spectrum Scale by performing the following substeps:
1. Issue the following command:
```
export PATH=$PATH:/usr/lpp/mmfs/bin
```
2. Create a nodes file. An example follows. For more information about the format of the file, see the description of the NodeFile option for the -N parameter of the mmcrcluster command.
```
-cat nodes 
bluhelix10:quorum-manager: 
bluhelix11::
bluhelix12::
```
3. Create the GPFS cluster by issuing the mmcrcluster command with the -N, -p, -s, -r, and -R parameters. An example follows:
```
mmcrcluster -N nodes -p bluhelix10 -s bluhelix12 -r /usr/bin/ssh -R /usr/bin/scp
```
4. Set the license mode for each node by issuing the mmchlicense command. Use a server or client license setting as appropriate. In the following example, all three nodes are servers:
```
mmchlicense server --accept -N bluhelix10 
mmchlicense server --accept -N bluhelix11 
mmchlicense server --accept -N bluhelix12
```
5. Set up the Network Shared Disks (NSDs):
  1. Determine the disk to use to create a file system by issuing the fdisk -l and df -Th commands.
  2. Find the storage device to use for Spectrum Scale. An example for iSCSI follows:
```
find /sys/devices/platform/host* -name block\* -exec ls -la '{}' \; | sed s#^.*../block/#/dev/#g
```
  3. Create the NSD file by using the storage device that you identified in the previous substep. An example follows; in this case, the storage device is called /dev/sda. For more information about the format of the NSD file, see the description of the -F parameter of the mmcrnsd command.
```
cat nsd
/dev/sda:bluhelix10,bluhelix11,bluhelix12::dataAndMetadata:1:::
```
  4. Create the NSDs by issuing the mmcrnsd command with the -F parameter. An example follows:
```
mmcrnsd -F nsd 
```
6. Start the cluster by issuing the following command:
```
mmstartup -a
```
7. Issue the following command and ensure that the output shows that all nodes are active:
```
mmgetstate -a -L
```
8. Using the NSDs that you created, create the cluster file system by issuing the mmcrfs command. An example follows. The parameters in the example are recommended, but ensure that they are appropriate for your setup.
```
mmcrfs clusterfs -F nsd -A yes -B 256K -M 3 -n 8 -R 3 -T /mnt/clusterfs
```
Mount the file system on all nodes by issuing the following command:
```
mmmount all -a
```