IBM Support

IBM Spectrum Scale cluster filesystem for Resiliency & Concurrent IO

Technical Blog Post


Abstract

IBM Spectrum Scale cluster filesystem for Resiliency & Concurrent IO

Body

image

IBM Spectrum Scale™, based on technology from IBM General Parallel File System (referred to as IBM Spectrum Scale or GPFS™), is a high performance shared-disk file management solution based on advanced clustering technologies that provides fast, reliable concurrent access to data from multiple servers without Single Point of Failure (SPoF).

GPFS are designed for scalability and performance with data intensive applications, where applications can readily access files using standard file system interfaces (POSIX), and the same file or directory can be accessed concurrently from multiple servers and protocols.

IBM Spectrum Scale is supported on AIX®, Linux and Windows Server operating systems. It is supported on IBM® POWER®, Intel or AMD Opteron based servers, and IBM Z.

 

Table of Content

  1. Start

  2. Stop

  3. Monitor

  4. Install & Configure

  5. Prerequisites

  6. References

Ensure that the PATH environment variable on each cluster node includes /usr/lpp/mmfs/bin, such as for KornShell: export PATH=$PATH:/usr/lpp/mmfs/bin

Start

To start the cluster services on all cluster nodes:

  # mmstartup -a

To start the cluster services on a specific cluster node:

  # mmstartup -N aixlpar2

To mount a cluster filesystem on all cluster nodes:

  # mmmount /shared -a  # mmmount clfs01 -a

Stop

To stop the cluster services on all nodes

  # mmshutdown -a

To stop the cluster services on a specific cluster node:

  # mmshutdown -N aixlpar2

To unmount a cluster filesystem on all cluster nodes:

  # mmumount /gpfs -a  # mmumount clfs01 -a

Monitor

To check the basic configuration and status of a cluster:

  # mmlscluster

To check the status of a cluster:

  # mmgetstate -aL

To check the cluster disks (NSDs) and corresponding AIX hdisk device names from one node (in this case aixlpar1)::

  # mmlsnsd -X

To check the usage of the cluster disks (NDSs) in a the cluster filesystem:

  # mmdf gpfsci01

Install & Configure

To create a basic dual AIX node GPFS cluster with SAN LUN hdisks directly accessible from all the nodes (up to 8):

  1. Verify prerequisites.
  2. Configure the AIX LPAR cluster nodes.
  3. Zone and mask the cluster shared SAN LUNs to all WWPNs for all the nodes.
  4. Install the Spectrum Scale (GPFS) Licensed Program Product (LPP).
  5. Create the cluster (from one node).
  6. Create the GPFS file system (from one node).
  7. Test and verify cluster functionality.

Verify prerequisites (on each node)

Firmware/microcode, device drivers, multi-path drivers, and other software levels and tunables as recommended by the storage vendor

  # lsmcode -A  # lslpp -l "*packagename*"  # lsdev -Cc disk -Fname|xargs -i lsattr -El {}

Configure the AIX LPAR cluster nodes

Consistent hostname resolution between nodes

Copy /etc/hosts from the second node to the primary node and run diff

  # diff /etc/hosts /tmp/hosts.node2

Copy /etc/netsvc.conf from the second node to the primary node and run diff (if /etc/resolv.conf is in place)

  # diff /etc/netsvc.conf /tmp/netsvc.conf.node2

Check that /etc/irs.conf is not used and that /etc/environment or root user environment do not have the NSORDER environment variable set.

Time synchronization between nodes (preferred)

If using NTP check the /etc/ntp.conf and xntpd:

  # lssrc -ls xntpd  # xntpdc sysinfo | sysstats | peers | listpeers

If not using NTP, setup periodic time sync between nodes with the ntpdate command

SSH between the nodes without password prompt or /etc/motd display from ssh login

If /etc/motd is used, create ~/.hushlogin: touch ~/.hushlogin

Exchange SSH public keys (old school:)

On all cluster nodes issue the following commands:

  # ssh-keygen -t rsa  # cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  # ssh root@<cluster node hostname>

Answer "yes" to add and only hit ENTER to all other questions, this last step will add the cluster node to itself.

On all cluster nodes issue the following commands:

  # cat ~/.ssh/id_rsa.pub| ssh root@<other cluster node> "cat >> ~/.ssh/authorized_keys"  # ssh root@<other cluster node hostname>

Answer "yes" to add and only hit ENTER to all other questions, this last step will add the other cluster node to the current cluster node.

Zone and mask the cluster shared SAN LUNs to all WWPNs for all the nodes

  • IF using NPIV, zone and mask to each LPARs NPIV adapters both WWPNs, for all cluster nodes on all Managed Systems
  • IF using VSCSI, zone and mask to each VIOS physical FC adapters WWPN, for all VIOS on all Managed Systems

Check that all the shared LUN hdisks devices are available on all cluster nodes (compare UUID):

  # lspv -u

Check that all the shared LUN hdisks devices have no_reserve set on all cluster nodes:

  # lsdev -Cc disk -Fname|xargs -i lsattr -Pl {} -a reservation_policy // check to ensure no_reserve is set on shared cluster disks  # lsdev -Cc disk -Fname|xargs -i chpv -Pl {} -a reservation_policy=no_reserve // change to no_reserve on shared cluster disks for next boot/load

Check that all the hdisks have no_reserve set on all cluster nodes:

  # lsdev -Cc disk -Fname|xargs -i devrsrv -c query -l {} // check current locking on shared cluster disks, ensure they are not locked

Install IBM Spectrum Scale™ LPPs (on each node)

Install LPPs for IBM Spectrum Scale™ Standard Edition:

  • gpfs.base

  • gpfs.docs.data

  • gpfs.msg.en_US

  • gpfs.license.xx

  • gpfs.compression

  • gpfs.gskit

  • gpfs.ext

  • gpfs.crypto (IBM Spectrum Scale™ Advanced Edition or IBM Spectrum Scale Data Management Edition only)

  • gpfs.adv (IBM Spectrum Scale Advanced Edition or IBM Spectrum Scale Data Management Edition only)

Check the installed version:

  # lslpp -L "gpfs.*"

Create the cluster (from one node)

Create the cluster (mmcrcluster)

  # mmcrcluster -C gpfscl1 -N aixlpar1:manager-quorum -p aixgpfs1 -r /usr/bin/ssh -R /usr/bin/scp  mmcrcluster: Processing node aixlpar1  mmcrcluster: Command successfully completed  mmcrcluster: Warning: Not all nodes have proper GPFS license designations.      Use the mmchlicense command to designate licenses as needed.

Enable the server license for server nodes (mmchlicense)

  # mmchlicense server --accept -N aixlpar1  The following nodes will be designated as possessing GPFS server licenses:          aixlpar1  mmchlicense: Command successfully completed

Start the cluster (mmstartup)

  # mmstartup -a  mmstartup: Starting GPFS ...

Check the status of the cluster (mmgetstate)

  # mmgetstate -a   Node number  Node name        GPFS state  ------------------------------------------         1      aixlpar1         active

Add the second node to the cluster (this can be done in one step while creating the cluster)

  # mmaddnode -N aixlpar2  mmaddnode: Processing node aixlpar2  mmaddnode: Command successfully completed  mmaddnode: Warning: Not all nodes have proper GPFS license designations.      Use the mmchlicense command to designate licenses as needed.  mmaddnode: Propagating the cluster configuration data to all    affected nodes.  This is an asynchronous process.

Enable the server license for the added server node

  # mmchlicense server --accept -N aixlpar2  The following nodes will be designated as possessing GPFS server licenses:          aixlpar2  mmchlicense: Command successfully completed  mmchlicense: Propagating the cluster configuration data to all    affected nodes.  This is an asynchronous process.

Start the cluster services on the added node

  # mmstartup -N aixlpar2  mmstartup: Starting GPFS ...

Change the cluster to add the new node as a secondary configuration server

  # mmchcluster -s aixlpar2  mmchcluster: GPFS cluster configuration servers:  mmchcluster:   Primary server:    aixlpar1  mmchcluster:   Secondary server:  aixlpar2  mmchcluster: Command successfully completed

Change the new node to a manager and quorum node

  # mmchnode --quorum --manager -N aixlpar2  mmchnode: Processing node aixlpar2  mmchnode: Propagating the cluster configuration data to all    affected nodes.  This is an asynchronous process.

Check the status of the cluster

  # mmgetstate -aL   Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state   Remarks  -------------------------------------------------------------------------------------         1      aixlpar1           1*        2          2       active       quorum node         2      aixlpar2           1*        2          2       active       quorum node

Check the basic configuration and status of the cluster

  # mmlscluster  GPFS cluster information  ========================    GPFS cluster name:         gpfscl1.aixgpfs1    GPFS cluster id:           11520425435678904690    GPFS UID domain:           gpfscl1.aixgpfs1    Remote shell command:      /usr/bin/ssh    Remote file copy command:  /usr/bin/scp    Repository type:           CCR     Node  Daemon node name  IP address    Admin node name  Designation  --------------------------------------------------------------------     1   aixlpar1          172.21.1.101  aixlpar1         quorum-manager     2   aixlpar2          172.21.1.102  aixlpar2         quorum-manager

Create a cluster file system

Format the cluster disks (NSDs) for cluster file systems (mmcrnsd) from one node

Create an NSD configuration file for the SAN LUN hdisks, such as gpfsdisks.txt

Input file BEFORE mmcrnsd processing

  %nsd:  device=hdisk7  Servers=aixlpar1,aixlpar2  %nsd:  device=hdisk8  Servers=aixlpar1,aixlpar2  %nsd:  device=hdisk9  Servers=aixlpar1,aixlpar2
  # mmcrnsd -F gpfsdisks.txt

Input file AFTER mmcrnsd processing

  %nsd: nsd=gpfs2nsd  device=hdisk7  Servers=aixlpar1,aixlpar2  %nsd:nsd=gpfs3nsd  device=hdisk8  Servers=aixlpar1,aixlpar2  %nsd: nsd=gpfs3nsd  device=hdisk9  Servers=aixlpar1,aixlpar2

Check the cluster disks (NSDs) to confirm they were converted (show show "free disk" on each)

  # mmlsnsd -d "gpfs2nsd;gpfs3nsd;gpfs4nsd"   File system   Disk name    NSD servers  ---------------------------------------------------------------------------   (free disk)   gpfs2nsd     aixlpar1,aixlpar2   (free disk)   gpfs3nsd     aixlpar1,aixlpar2   (free disk)   gpfs4nsd     aixlpar1,aixlpar2

Update the cluster with quorum tiebreaker NSDs, can be performed online from GPFS v4.1 with cluster configuration repository (CCR), otherwise shutdown the cluster, add the tiebreakers and start the cluster

  # mmchconfig tiebreakerdisks="gpfs2nsd,gpfs3nsd,gpfs4nsd"

Create the GPFS file system (mmcrfs), in this case calling it clfs01 with mountpoint /shared

  # mmcrfs /shared clfs01 -F gpfsdisks.txt -A yes

Check the cluster disks (NSDs) to confirm they are allocated (show show the filesystem name on each)

  # mmlsnsd -d "gpfs2nsd;gpfs3nsd;gpfs4nsd"   File system   Disk name    NSD servers  ---------------------------------------------------------------------------   clfs01        gpfs2nsd     aixlpar1,aixlpar2   clfs01        gpfs3nsd     aixlpar1,aixlpar2   clfs01        gpfs4nsd     aixlpar1,aixlpar2

Mount the file system (mmmount) on all cluster nodes

  # mmmount clfs01 -a

Check the usage of the cluster disks (NDSs) in a the filesystem

  # mmdf clfs01  disk                disk size  failure holds    holds           free in KB          free in KB  name                    in KB    group metadata data        in full blocks        in fragments  --------------- ------------- -------- -------- ----- -------------------- -------------------  Disks in storage pool: system (Maximum disk size allowed is 768 GB)  gpfs2nsd             52428800       -1 yes      yes        51765248 ( 99%)         11384 ( 0%)  gpfs3nsd             52428800       -1 yes      yes        51798016 ( 99%)         11128 ( 0%)  gpfs4nsd             52428800       -1 yes      yes        51773440 ( 99%)         11128 ( 0%)                  -------------                         -------------------- -------------------  (pool total)        157286400                             155336704 ( 99%)         33640 ( 0%)                    =============                         ==================== ===================  (total)             157286400                             155336704 ( 99%)         33640 ( 0%)    Inode Information  -----------------  Number of used inodes:            4041  Number of free inodes:          152631  Number of allocated inodes:     156672  Maximum number of inodes:       156672

Prerequisites

For Spectrum Scale go to https://www.ibm.com/mysupport/s/topic/0TO50000000QW2fGAG/spectrum-scale

  1. Select "Download fixes, drivers and firmware"
  2. Select the tab for one of: Advanced, Data Management, Express or Standard
  3. In the "Search:" box, type "AIX

For AIX and Spectrum Scale (or GPFS v4.1 and previous) go to Fix Level Recommendation Tool (FLRT) at http://www14.software.ibm.com/webapp/set2/flrt/power

  • Choose System Type

Choose System Specifications

Server Machine-Type-Model

System firmware

HMC and HMC Virtual Appliance

  • Partition

Select OS: AIX

Select version of AIX

Select "Spectrum Scale" at the bottom (for GPFS v4.1 and previous select the edition of GPFS)

Select version of Spectrum Scale (for GPFS v4.1 and previous do the same)

  • Click "SUBMIT"
  • Review the "IBM FLRT Power System Report"

References

IBM Spectrum Scale

Administering

Configuring

Monitoring

Troubleshooting

Command reference

Creating immutable filesets and files

IBM Spectrum Scale Software Version Recommendation Preventive Service Planning

IBM Spectrum Scale - Supported Upgrade Paths

IBM Spectrum Scale - Functional Support Matrices

General Parallel File System V4.1

IBM Spectrum Scale™ Frequently Asked Questions and Answers

 

 

[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"HW206","label":"Storage Systems"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

UID

ibm16165117

Page Feedback