IBM Spectrum Scale cluster filesystem for Resiliency & Concurrent IO

News

Abstract

By Bjorn Roden, IBM Systems Lab Services

IBM Spectrum Scale™, based on technology from IBM General Parallel File System (referred to as IBM Spectrum Scale or GPFS™), is a high performance shared-disk file management solution based on advanced clustering technologies that provides fast, reliable concurrent access to data from multiple servers without Single Point of Failure (SPoF).

GPFS are designed for scalability and performance with data intensive applications, where applications can readily access files using standard file system interfaces (POSIX), and the same file or directory can be accessed concurrently from multiple servers and protocols.

IBM Spectrum Scale is supported on AIX®, Linux and Windows Server operating systems. It is supported on IBM® POWER®, Intel or AMD Opteron based servers, and IBM Z.

Content

Table of Content

Start
Stop
Monitor
Install & Configure
Prerequisites
References

Ensure that the PATH environment variable on each cluster node includes /usr/lpp/mmfs/bin, such as for KornShell: export PATH=$PATH:/usr/lpp/mmfs/bin

Start

To start the cluster services on all cluster nodes:

# mmstartup -a

To start the cluster services on a specific cluster node:

# mmstartup -N aixlpar2

To mount a cluster filesystem on all cluster nodes:

# mmmount /shared -a

# mmmount clfs01 -a

Stop

To stop the cluster services on all nodes

# mmshutdown -a

To stop the cluster services on a specific cluster node:

# mmshutdown -N aixlpar2

To unmount a cluster filesystem on all cluster nodes:

# mmumount /gpfs -a

# mmumount clfs01 -a

Monitor

To check the basic configuration and status of a cluster:

# mmlscluster

To check the status of a cluster:

# mmgetstate -aL

To check the cluster disks (NSDs) and corresponding AIX hdisk device names from one node (in this case aixlpar1)::

# mmlsnsd -X

To check the usage of the cluster disks (NDSs) in a the cluster filesystem:

# mmdf gpfsci01

Install & Configure

To create a basic dual AIX node GPFS cluster with SAN LUN hdisks directly accessible from all the nodes (up to 8):

Verify prerequisites.
Configure the AIX LPAR cluster nodes.
Zone and mask the cluster shared SAN LUNs to all WWPNs for all the nodes.
Install the Spectrum Scale (GPFS) Licensed Program Product (LPP).
Create the cluster (from one node).
Create the GPFS file system (from one node).
Test and verify cluster functionality.

Verify prerequisites (on each node)

Firmware/microcode, device drivers, multi-path drivers, and other software levels and tunables as recommended by the storage vendor

# lsmcode -A

# lslpp -l "*packagename*"

# lsdev -Cc disk -Fname|xargs -i lsattr -El {}

Configure the AIX LPAR cluster nodes

Consistent hostname resolution between nodes

Copy /etc/hosts from the second node to the primary node and run diff

# diff /etc/hosts /tmp/hosts.node2

Copy /etc/netsvc.conf from the second node to the primary node and run diff (if /etc/resolv.conf is in place)

# diff /etc/netsvc.conf /tmp/netsvc.conf.node2

Check that /etc/irs.conf is not used and that /etc/environment or root user environment do not have the NSORDER environment variable set.

Time synchronization between nodes (preferred)

If using NTP check the /etc/ntp.conf and xntpd:

# lssrc -ls xntpd

# xntpdc sysinfo | sysstats | peers | listpeers

If not using NTP, setup periodic time sync between nodes with the ntpdate command

SSH between the nodes without password prompt or /etc/motd display from ssh login

If /etc/motd is used, create ~/.hushlogin: touch ~/.hushlogin

Exchange SSH public keys (old school:)

On all cluster nodes issue the following commands:

# ssh-keygen -t rsa

# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

# ssh root@<cluster node hostname>

Answer "yes" to add and only hit ENTER to all other questions, this last step will add the cluster node to itself.

On all cluster nodes issue the following commands:

# cat ~/.ssh/id_rsa.pub| ssh root@<other cluster node> "cat >> ~/.ssh/authorized_keys"

# ssh root@<other cluster node hostname>

Answer "yes" to add and only hit ENTER to all other questions, this last step will add the other cluster node to the current cluster node.

Zone and mask the cluster shared SAN LUNs to all WWPNs for all the nodes

IF using NPIV, zone and mask to each LPARs NPIV adapters both WWPNs, for all cluster nodes on all Managed Systems
IF using VSCSI, zone and mask to each VIOS physical FC adapters WWPN, for all VIOS on all Managed Systems

Check that all the shared LUN hdisks devices are available on all cluster nodes (compare UUID):

# lspv -u

Check that all the shared LUN hdisks devices have no_reserve set on all cluster nodes:

# lsdev -Cc disk -Fname|xargs -i lsattr -Pl {} -a reservation_policy // check to ensure no_reserve is set on shared cluster disks

# lsdev -Cc disk -Fname|xargs -i chpv -Pl {} -a reservation_policy=no_reserve // change to no_reserve on shared cluster disks for next boot/load

Check that all the hdisks have no_reserve set on all cluster nodes:

# lsdev -Cc disk -Fname|xargs -i devrsrv -c query -l {} // check current locking on shared cluster disks, ensure they are not locked

Install IBM Spectrum Scale™ LPPs (on each node)

Install LPPs for IBM Spectrum Scale™ Standard Edition:

gpfs.base
gpfs.docs.data
gpfs.msg.en_US
gpfs.license.xx
gpfs.compression
gpfs.gskit
gpfs.ext
gpfs.crypto (IBM Spectrum Scale™ Advanced Edition or IBM Spectrum Scale Data Management Edition only)
gpfs.adv (IBM Spectrum Scale Advanced Edition or IBM Spectrum Scale Data Management Edition only)

Check the installed version:

# lslpp -L "gpfs.*"

Create the cluster (from one node)

Create the cluster (mmcrcluster)

# mmcrcluster -C gpfscl1 -N aixlpar1:manager-quorum -p aixgpfs1 -r /usr/bin/ssh -R /usr/bin/scp

mmcrcluster: Processing node aixlpar1

mmcrcluster: Command successfully completed

mmcrcluster: Warning: Not all nodes have proper GPFS license designations.

    Use the mmchlicense command to designate licenses as needed.

Enable the server license for server nodes (mmchlicense)

# mmchlicense server --accept -N aixlpar1

The following nodes will be designated as possessing GPFS server licenses:

        aixlpar1

mmchlicense: Command successfully completed

Start the cluster (mmstartup)

# mmstartup -a

mmstartup: Starting GPFS ...

Check the status of the cluster (mmgetstate)

# mmgetstate -a

 Node number  Node name        GPFS state

------------------------------------------

       1      aixlpar1         active

Add the second node to the cluster (this can be done in one step while creating the cluster)

# mmaddnode -N aixlpar2

mmaddnode: Processing node aixlpar2

mmaddnode: Command successfully completed

mmaddnode: Warning: Not all nodes have proper GPFS license designations.

    Use the mmchlicense command to designate licenses as needed.

mmaddnode: Propagating the cluster configuration data to all

  affected nodes.  This is an asynchronous process.

Enable the server license for the added server node

# mmchlicense server --accept -N aixlpar2

The following nodes will be designated as possessing GPFS server licenses:

        aixlpar2

mmchlicense: Command successfully completed

mmchlicense: Propagating the cluster configuration data to all

  affected nodes.  This is an asynchronous process.

Start the cluster services on the added node

# mmstartup -N aixlpar2

mmstartup: Starting GPFS ...

Change the cluster to add the new node as a secondary configuration server

# mmchcluster -s aixlpar2

mmchcluster: GPFS cluster configuration servers:

mmchcluster:   Primary server:    aixlpar1

mmchcluster:   Secondary server:  aixlpar2

mmchcluster: Command successfully completed

Change the new node to a manager and quorum node

# mmchnode --quorum --manager -N aixlpar2

mmchnode: Processing node aixlpar2

mmchnode: Propagating the cluster configuration data to all

  affected nodes.  This is an asynchronous process.

Check the status of the cluster

# mmgetstate -aL

 Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state   Remarks

-------------------------------------------------------------------------------------

       1      aixlpar1           1*        2          2       active       quorum node

       2      aixlpar2           1*        2          2       active       quorum node

Check the basic configuration and status of the cluster

# mmlscluster

GPFS cluster information

========================

  GPFS cluster name:         gpfscl1.aixgpfs1

  GPFS cluster id:           11520425435678904690

  GPFS UID domain:           gpfscl1.aixgpfs1

  Remote shell command:      /usr/bin/ssh

  Remote file copy command:  /usr/bin/scp

  Repository type:           CCR

 Node  Daemon node name  IP address    Admin node name  Designation

--------------------------------------------------------------------

   1   aixlpar1          172.21.1.101  aixlpar1         quorum-manager

   2   aixlpar2          172.21.1.102  aixlpar2         quorum-manager

Create a cluster file system

Format the cluster disks (NSDs) for cluster file systems (mmcrnsd) from one node

Create an NSD configuration file for the SAN LUN hdisks, such as gpfsdisks.txt

Input file BEFORE mmcrnsd processing

%nsd:

device=hdisk7

Servers=aixlpar1,aixlpar2

%nsd:

device=hdisk8

Servers=aixlpar1,aixlpar2

%nsd:

device=hdisk9

Servers=aixlpar1,aixlpar2

# mmcrnsd -F gpfsdisks.txt

Input file AFTER mmcrnsd processing

%nsd: nsd=gpfs2nsd

device=hdisk7

Servers=aixlpar1,aixlpar2

%nsd:nsd=gpfs3nsd

device=hdisk8

Servers=aixlpar1,aixlpar2

%nsd: nsd=gpfs3nsd

device=hdisk9

Servers=aixlpar1,aixlpar2

Check the cluster disks (NSDs) to confirm they were converted (show show "free disk" on each)

# mmlsnsd -d "gpfs2nsd;gpfs3nsd;gpfs4nsd"

 File system   Disk name    NSD servers

---------------------------------------------------------------------------

 (free disk)   gpfs2nsd     aixlpar1,aixlpar2

 (free disk)   gpfs3nsd     aixlpar1,aixlpar2

 (free disk)   gpfs4nsd     aixlpar1,aixlpar2

Update the cluster with quorum tiebreaker NSDs, can be performed online from GPFS v4.1 with cluster configuration repository (CCR), otherwise shutdown the cluster, add the tiebreakers and start the cluster

# mmchconfig tiebreakerdisks="gpfs2nsd,gpfs3nsd,gpfs4nsd"

Create the GPFS file system (mmcrfs), in this case calling it clfs01 with mountpoint /shared

# mmcrfs /shared clfs01 -F gpfsdisks.txt -A yes

Check the cluster disks (NSDs) to confirm they are allocated (show show the filesystem name on each)

# mmlsnsd -d "gpfs2nsd;gpfs3nsd;gpfs4nsd"

 File system   Disk name    NSD servers

---------------------------------------------------------------------------

 clfs01        gpfs2nsd     aixlpar1,aixlpar2

 clfs01        gpfs3nsd     aixlpar1,aixlpar2

 clfs01        gpfs4nsd     aixlpar1,aixlpar2

Mount the file system (mmmount) on all cluster nodes

# mmmount clfs01 -a

Check the usage of the cluster disks (NDSs) in a the filesystem

# mmdf clfs01

disk                disk size  failure holds    holds           free in KB          free in KB

name                    in KB    group metadata data        in full blocks        in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 768 GB)

gpfs2nsd             52428800       -1 yes      yes        51765248 ( 99%)         11384 ( 0%)

gpfs3nsd             52428800       -1 yes      yes        51798016 ( 99%)         11128 ( 0%)

gpfs4nsd             52428800       -1 yes      yes        51773440 ( 99%)         11128 ( 0%)

                -------------                         -------------------- -------------------

(pool total)        157286400                             155336704 ( 99%)         33640 ( 0%)

                =============                         ==================== ===================

(total)             157286400                             155336704 ( 99%)         33640 ( 0%)

Inode Information

-----------------

Number of used inodes:            4041

Number of free inodes:          152631

Number of allocated inodes:     156672

Maximum number of inodes:       156672

Prerequisites

For Spectrum Scale go to https://www.ibm.com/mysupport/s/topic/0TO50000000QW2fGAG/spectrum-scale

Select "Download fixes, drivers and firmware"
Select the tab for one of: Advanced, Data Management, Express or Standard
In the "Search:" box, type "AIX

For AIX and Spectrum Scale (or GPFS v4.1 and previous) go to Fix Level Recommendation Tool (FLRT) at http://www14.software.ibm.com/webapp/set2/flrt/power

Choose System Type

Choose System Specifications

Server Machine-Type-Model

System firmware

HMC and HMC Virtual Appliance

Partition

Select OS: AIX

Select version of AIX

Select "Spectrum Scale" at the bottom (for GPFS v4.1 and previous select the edition of GPFS)

Select version of Spectrum Scale (for GPFS v4.1 and previous do the same)

Click "SUBMIT"
Review the "IBM FLRT Power System Report"

References

IBM Spectrum Scale

https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/ibmspectrumscale501_welcome.html

Administering

https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adm_administering_kclanding.htm

Configuring

https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adm_configuring_kclanding.htm

Monitoring

https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adv_monitoring_kclanding.htm

Troubleshooting

https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1pdg_troubleshooting_kclanding.htm

Command reference

https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adm_command.htm

Creating immutable filesets and files

https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.1/com.ibm.spectrum.scale.v5r01.doc/bl1adm_createimmutablefilesets.htm

IBM Spectrum Scale Software Version Recommendation Preventive Service Planning

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009703

IBM Spectrum Scale - Supported Upgrade Paths

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#upgrade_paths

IBM Spectrum Scale - Functional Support Matrices

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#fsm

General Parallel File System V4.1

https://www.ibm.com/support/knowledgecenter/en/SSFKCN_4.1.0/gpfs.v4r1_welcome.html

IBM Spectrum Scale™ Frequently Asked Questions and Answers

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#gpfsadv

Related Information

IBM Systems Lab Services

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STXKQY","label":"IBM Spectrum Scale"},"Component":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Tips