Cleanup procedures required if reinstalling with the installation toolkit

Before you can reinstall with the installation toolkit, you must perform some cleanup procedures first.

You can use the following procedures to clean up various stages of the previous GPFS installation and protocols deployment.

Starting the cleanup process

  1. Clean up the installer directory by issuing the following command:
    mmdsh -N NodeList rm -rf /usr/lpp/mmfs/5.1.3.x
    Note: The installer directory in this command depends upon the release version.
  2. Check /etc/resolv.conf for any changes that pointed the authentication directory to DNS, and remove them.
  3. If the object protocol was enabled and storage policies were created, use the mmobj policy list -v command and save the output list before you run the mmces service disable OBJ command.

Uninstalling Ansible

The installation toolkit is built on the Ansible® automation platform. For more information, see Supported Ansible version.

You might need to uninstall Ansible as a part of the cleanup process or if the supported version is not installed. Check the Ansible version that is installed and, if needed, uninstall Ansible after verifying that it is not being used for any other purpose.
  1. Check the version of Ansible that is installed by using one of the following commands.
    ansible --version

    or

    rpm -qa | grep ansible
  2. Uninstall Ansible by using one of the following commands depending on your environment.
    • Uninstall by using the rpm command on RHEL and SLES nodes.
      rpm -qa | grep ansible | xargs rpm -e
    • Uninstall by using the yum command on RHEL nodes.
      yum remove ansible
    • Uninstall by using the pip3 command on RHEL, SLES, and Ubuntu nodes.
      pip3 uninstall ansible
    • Uninstall by using the apt-get command on Ubuntu nodes.
      apt-get purge ansible

Cleaning up the authentication configurations

Use these commands to completely remove the current authentication configuration for all protocols, NFS/SMB only, or Object only. Be aware that doing so may cause loss of access to previously written data.

  1. Remove the configurations. To do so, issue the following commands:
    mmuserauth service remove --data-access-method file
mmuserauth service remove --data-access-method object
  2. Delete the ID-mapping information when authentication was configured with AD. To do so, use the following command.
    Note: Once this ID-mapping information is deleted, you might not be able to access existing files (or you might experience other unforeseen access issues), so do this only if you do not need access to existing data.
    mmuserauth service remove --data-access-method file --idmapdelete

Disabling CES

Disable CES on every node nodeNameX. To do so, issue the following command:
mmchnode -N nodeNameX --ces-disable

Protocol cleanup steps

  1. Use the following steps to clean up some of the object state on the system:
    mmdsh -N cesNodes systemctl stop postgresql-obj
rm -rf path_to_cesSharedRoot/object/keystone
    Delete the objectization temp directory. In the following example, the temp directory was created in the fs1 file system at /ibm/fs1/ibmobjectizer/tmp:
    rm -rf  /ibm/fs1/ibmobjectizer/tmp
  2. Remove the fileset created for object.

    In the following example, the configured fileset object_fileset in file system fs1 was linked at /gpfs/fs1.

    mmlsfileset fs1
mmunlinkfileset fs1 object_fileset
mmdelfileset fs1 object_fileset -f
  3. Remove any fileset created for an object storage policy.

    Run the mmobj policy list -v command. If for some reason mmobj policy list -v cannot be run (For example, if the Object Protocol or CES was already disabled) or you cannot detect which filesets are used for object storage policies, contact the IBM® Support Center.

    For example:

    If the mmobj policy list command returned the following and the filesets got created in file system fs1:

    Index       Name           Deprecated       Fileset Path                       Functions              Function Details
                           Fileset            							    
------------------------------------------------------------------------------------------------------------------------
0           SwiftDefault  object_fileset   /ibm/cesSharedRoot/object_fileset
11751509160 sof-policy1   obj_sof-policy1  /ibm/cesSharedRoot/obj_sof-policy1  file-and-object-access regions="1"
11751509230 mysofpolicy   obj_mysofpolicy  /ibm/cesSharedRoot/obj_mysofpolicy  file-and-object-access regions="1"
11751510260 Test19        obj_Test19       /ibm/cesSharedRoot/obj_Test19                              regions="1"

    Use the following commands for each row except the first:

    mmunlinkfileset fs1 sof-policy1
mmdelfileset fs1 sof-policy1 -f
  4. Clear the cesSharedRoot configuration by issuing the following command:
    mmchconfig cesSharedRoot=DEFAULT
  5. Check your recent package installs to determine which packages need to be removed by issuing the following commands:
    RHEL and SLES
    rpm -qa --last|more
    Ubuntu
    dpkg -l
    After you have determined which packages need to be removed, do so by issuing the following commands, as needed:
    RHEL and SLES
    mmdsh -N NodeList yum erase gpfs.smb nfs-ganesha-mount \
nfs-ganesha-vfs PyQt4 nfs-ganesha-utils phonon-backend-gstreamer phonon sip qt-x11 kde-filesystem \
qt qt-settings libmng nfs-ganesha-proxy nfs-ganesha-nullfs nfs-ganesha-gpfs  -y

mmdsh -N NodeList yum erase python-webob MySQL-python mariadb-server perl-DBD-MySQL mariadb \
python-futures python-warlock python-jsonpatch python-jsonschema python-jsonpointer python-cmd2 \
python-cliff pyparsing memcached python-oslo-utils python-oslo-serialization python-oslo-i18n \
python-babel babel python-keyring python-stevedore python-prettytable python-pbr python-oslo-config \
python-netaddr python-iso8601 python-dnspython python-urllib3 python-six python-requests \
python-paste-deploy python-tempita python-paste python-netifaces python-simplejson \
python-greenlet python-eventlet -y

mmdsh -N NodeList yum erase python2-keystoneauth1 python2-keystoneclient python2-keystonemiddleware \
python2-openstackclient python2-oslo-cache python2-oslo-concurrency python2-oslo-config \
python2-oslo-context python2-oslo-db python2-oslo-i18n python2-oslo-middleware python2-oslo-policy \
python2-oslo-service python2-swiftclient python3-keystone python3-keystoneauth1 python3-keystoneclient \
python3-keystonemiddleware python3-openstackclient python3-openstacksdk python3-oslo-cache \
python3-oslo-concurrency python3-oslo-config python3-oslo-context python3-oslo-db python3-oslo-i18n \
python3-oslo-log python3-oslo-messaging python3-oslo-middleware python3-oslo-policy \
python3-oslo-serialization python3-oslo-service python3-oslo-upgradecheck python3-oslo-utils \
python3-swift python3-swiftclient python-openstackclient-lang python-openstacksdk \
python-oslo-cache-lang python-oslo-concurrency-lang python-oslo-db-lang python-oslo-i18n-lang \
python-oslo-log python-oslo-log-lang python-oslo-messaging python-oslo-middleware-lang \
python-oslo-policy-lang python-oslo-utils-lang python-swift -y


mmdsh -N NodeList yum erase gpfs.smb-debugsource -y

mmdsh -N NodeList yum erase sssd-tools krb5-workstation

mmdsh -N NodeList yum erase python-routes python-repoze-lru python-oauthlib python-crypto \
python-qpid-common python-qpid python-dogpile-cache python-fixtures python-dogpile-core \
python-testtools python-extras python-oslo-context python-kombu python-anyjson python-amqp \
python-mimeparse python-markupsafe PyYAML python-passlib libyaml python-ibm-db-sa python-ibm-db \
python-sqlparse python-memcached python-posix_ipc python-sqlalchemy -y

mmdsh -N NodeList yum erase gpfs.gss.pmsensors gpfs.gss.pmcollector gpfs.pm-ganesha boost-regex -y

mmdsh -N NodeList yum erase openstack-utils -y

mmdsh -N NodeList yum erase python-ordereddict -y

mmdsh -N NodeList yum erase mariadb-libs -y

mmdsh -N NodeList yum erase pmswift -y

mmdsh -N NodeList yum erase python-cryptography python-enum34 swift3 python-pyeclib liberasurecode \
openstack-utils crudini python-msgpack python-wsgiref python-pycparser python-ply python-cffi \
openstack-selinux xmlsec1 python-pyasn1 python-retrying \
postgresql-server postgresql python-psycopg2 python-repoze-who jerasure gf-complete \
python-zope-interface postgresql-libs pytz -y

mmdsh -N NodeList yum erase spectrum-scale-object spectrum-scale-object-selinux \
openstack-swift openstack-swift-account openstack-swift-container openstack-swift-object \
openstack-swift-proxy python-swiftclient swiftonfile keystonemiddleware openstack-keystone \
python-keystone python-cinderclient python-glanceclient python-keystoneclient python-neutronclient \
python-novaclient python-keystoneauth1 python-openstackclient -y
    Ubuntu
    mmdsh -N NodeList dpkg -P gpfs.smb nfs-ganesha-mount nfs-ganesha-vfs \
PyQt4 nfs-ganesha-utils phonon-backend-gstreamer phonon sip qt-x11 kde-filesystem qt qt-settings libmng \
nfs-ganesha-proxy nfs-ganesha-nullfs gpfs.nfs-ganesha-gpfs gpfs.nfs-ganesha  -y

mmdsh -N NodeList dpkg -P python-webob MySQL-python mariadb-server perl-DBD-MySQL mariadb \
python-futures python-warlock python-jsonpatch python-jsonschema python-jsonpointer python-cmd2 \
python-cliff pyparsing memcached python-oslo-utils python-oslo-serialization python-oslo-i18n \
python-babel babel python-keyring python-stevedore python-prettytable python-pbr python-oslo-config \
python-netaddr python-iso8601 python-dnspython python-urllib3 python-six python-requests \
python-paste-deploy python-tempita python-paste python-netifaces python-simplejson \
python-greenlet python-eventlet -y

mmdsh -N NodeList dpkg -P python-routes python-repoze-lru python-oauthlib python-crypto \
python-qpid-common python-qpid python-dogpile-cache python-fixtures python-dogpile-core \
python-testtools python-extras python-oslo-context python-kombu python-anyjson python-amqp \
python-mimeparse python-markupsafe PyYAML python-passlib libyaml python-ibm-db-sa python-ibm-db \
python-sqlparse python-memcached python-posix_ipc python-sqlalchemy -y

mmdsh -N NodeList dpkg -P gpfs.gss.pmsensors gpfs.gss.pmcollector gpfs.pm-ganesha boost-regex -y

mmdsh -N NodeList dpkg -P openstack-utils gpfs.gpfs.nfs-ganesha-gpfs -y

mmdsh -N NodeList dpkg -P python-ordereddict -y

mmdsh -N NodeList dpkg -P mariadb-libs -y

mmdsh -N NodeList dpkg -P pmswift -y

mmdsh -N NodeList dpkg -P python-cryptography python-enum34 swift3 python-pyeclib liberasurecode \
crudini python-msgpack python-wsgiref python-pycparser python-ply python-cffi \
openstack-selinux xmlsec1 python-pyasn1 python-retrying postgresql-server postgresql \
python-psycopg2 python-repoze-who jerasure gf-complete python-zope-interface postgresql-libs \
pytz -y

mmdsh -N NodeList dpkg -P spectrum-scale-object spectrum-scale-object-selinux \
openstack-swift openstack-swift-account openstack-swift-container openstack-swift-object \
openstack-swift-proxy python-swiftclient swiftonfile keystonemiddleware openstack-keystone \
python-keystone python-keystoneauth1 python-cinderclient python-glanceclient python-keystoneclient \
python-neutronclient python-novaclient python-openstackclient -y

    File protocols authentication packages cleanup

    RHEL
    mmdsh -N NodeList yum erase bind-utils krb5-workstation openldap-clients sssd-tools ypbind yp-tools -y
    SLES
    mmdsh -N NodeList yum erase bind-utils krb5-client openldap2-client sssd-tools ypbind yp-tools -y
    Ubuntu
    mmdsh -N NodeList dpkg -P dnsutils krb5-user ldap-utils sssd-tools nis libslp1 -y
    Note: Plan to erase the SSSD packages only if you do not plan to use the SSSD service on the nodes for any other purpose.

    SSSD packages cleanup commands are as follows:

    RHEL
    mmdsh -N NodeList yum erase sssd*
    SLES
    mmdsh -N NodeList yum erase sssd*
    Ubuntu
    mmdsh -N NodeList dpkg -P sssd*
  6. Issue rpm -qa --last|more or dpkg -l on all nodes again, as the preceding list does not contain everything in the new builds.

    If the pmswift package still exists, remove it as follows:

    RHEL and SLES
    mmdsh -N NodeList rpm -e pmswift-5.1.3-x.noarch --noscripts
    Ubuntu
    mmdsh -N NodeList dpkg -P pmswift-5.1.3-x.noarch --noscripts

    You might also need to remove gpfs.smb.

  7. Clean up performance monitoring tool files. To do so, issue the following commands:
    mmdsh -N NodeList rm -rf /opt/IBM/zimon*
mmdsh -N NodeList rm -rf /usr/IBM/zimon*
mmdsh -N NodeList rm -rf /var/log/cnlog/zimon*
mmdsh -N NodeList rm -rf /var/lib/yum/repose/x86_64/7Server/*zimon*
mmdsh -N NodeList rm -rf /var/lib/yum/repose/ppc64/7Server/*zimon*
  8. Clean up CTDB. To do so, issue the following commands:
    mmdsh -N NodeList rm -rf /var/lib/ctdb
  9. Clean up Swift files. To do so, issue the following commands:
    mmdsh -N NodeList rm -rf /etc/swift
mmdsh -N NodeList rm -rf /var/lib/mysql
mmdsh -N NodeList rm -rf /etc/keystone
mmdsh -N NodeList rm -rf /var/lib/keystone
mmdsh -N NodeList rm -rf /root/openrc
mmdsh -N NodeList rm -rf /var/cache/yum/x86_64/7Server/*
mmdsh -N NodeList rm -rf /var/cache/yum/ppc64/7Server/*
mmdsh -N NodeList rm -rf /var/log/maria*
mmdsh -N NodeList rm -rf /usr/share/pixmaps/comps/maria*
mmdsh -N NodeList rm -rf /var/log/keystone
mmdsh -N NodeList rm -rf /var/spool/cron/keystone
mmdsh -N NodeList rm -rf /usr/lib/python2.7/site-packages/sos/plugins/openstack_keystone*
mmdsh -N NodeList rm -rf /tmp/keystone-signing-swift
mmdsh -N NodeList rm -rf /usr/lib/python2.7/site-packages/sos/plugins/*
mmdsh -N NodeList rm -rf /var/lib/yum/repos/x86_64/7Server/icm_openstack
mmdsh -N NodeList rm -f /usr/lib/python2.7/site-packages/swiftonfile-2.5.0_0-py2.7.egg*
mmdsh -N NodeList rm -f /usr/bin/*objectizer*.pyc
mmdsh -N NodeList rm -f /usr/bin/generate_dbmap.pyc
mmdsh -N NodeList systemctl disable openstack-keystone.service
mmdsh -N NodeList systemctl disable openstack-swift-container-updater.service
mmdsh -N NodeList systemctl disable openstack-swift-container-update
mmdsh -N NodeList systemctl disable openstack-swift-object-updater.service
mmdsh -N NodeList systemctl disable openstack-swift-container-auditor.service
mmdsh -N NodeList systemctl disable openstack-swift-container-replicator.service
mmdsh -N NodeList systemctl disable openstack-swift-container.service
mmdsh -N NodeList systemctl disable openstack-swift-object-replicator.service
mmdsh -N NodeList systemctl disable openstack-swift-object.service
mmdsh -N NodeList systemctl disable openstack-keystone.service
mmdsh -N NodeList systemctl disable openstack-swift-account-reaper.service
mmdsh -N NodeList systemctl disable openstack-swift-account-auditor.service
mmdsh -N NodeList systemctl disable openstack-swift-account-replicator.service
mmdsh -N NodeList systemctl disable openstack-swift-account.service
mmdsh -N NodeList systemctl disable openstack-swift-proxy.service
mmdsh -N NodeList systemctl disable openstack-swift-object-auditor.service
mmdsh -N NodeList systemctl disable openstack-swift-object-expirer.service
mmdsh -N NodeList systemctl disable openstack-swift-container-reconciler.service
mmdsh -N NodeList rm -rf
/var/lib/yum/yumdb/o/0b01eb65826df92befd8c161798cb842fa3c941e-openstack-utils-2015.1-201502031913.ibm.el7.7-noarch
  10. Reboot the nodes. To do so, issue the following command:
    mmdsh -N NodeList shutdown -r now
    If some nodes do not fully come up, do the following:
    1. Power off that node.
    2. Wait for a minute, then power on the node back.
      Note: Some nodes might need to be powered off and on more than once.
  11. Clean up Yum on all nodes. To do so, issue the following commands:
    mmdsh -N NodeList yum clean all
mmdsh -N NodeList rm -rf /etc/yum.repos.d/gpfs.repo /etc/yum.repos.d/icm* /etc/yum.repos.d/ces.repo 
mmdsh -N NodeList rm -rf /etc/yum.repos.d/epel* /etc/yum.repos.d/rdo*
mmdsh -N NodeList rm -rf /var/lib/yum/repos/x86_64/7Server/*
mmdsh -N NodeList rm -rf /var/lib/yum/repos/ppc64/7Server/*
  12. Remove GPFS.
    Notes:
    • This step is not required if you know that GPFS has not changed between old and new builds.
    • This step is not required if you prefer to perform an upgrade of GPFS.
    • This step is not required if you would like to keep the base GPFS installation intact and merely rerun the protocols deployment.
    • (To permanently remove GPFS, see Steps to permanently uninstall IBM Spectrum Scale.)
    1. Check which GPFS packages are on each node by issuing the following command:
      RHEL and SLES
      mmdsh -N NodeList rpm -qa|grep gpfs
      Ubuntu
      mmdsh -N NodeList dpkg -l|grep gpfs
      The system displays output similar to the following: 
      gpfs.msg.en_US-5.1.3-x.noarch
gpfs.gskit-8.0.55.x.x86_64
gpfs.license.xx-5.1.3-x.x86_64
gpfs.crypto-5.1.3-x.x86_64
gpfs.adv-5.1.3-x.x86_64
gpfs.docs-5.1.3-x.noarch
gpfs.base-5.1.3-x.x86_64
gpfs.gpl-5.1.3-x.noarch
    2. Before removing anything, make sure that GPFS is shut down on all nodes. To do so, issue the following command:
      mmshutdown -a
    3. Remove the packages by issuing the following commands in the order shown.
      Note: When you remove the gpfs.base package, you lose the mmdsh access.
      Therefore, be sure to remove gpfs.base last, as shown here.
      RHEL and SLES
      
mmdsh -N NodeList rpm -e gpfs.base-debuginfo-5.1.3-x.x86_64
mmdsh -N NodeList rpm -e gpfs.crypto-5.1.3-x.x86_64
mmdsh -N NodeList rpm -e gpfs.adv-5.1.3-x.x86_64
mmdsh -N NodeList rpm -e gpfs.msg.en_US-5.1.3-x.noarch
mmdsh -N NodeList rpm -e gpfs.gskit-8.0.55.x.x86_64
mmdsh -N NodeList rpm -e gpfs.compression-5.1.3-x.x86_64
mmdsh -N NodeList rpm -e gpfs.license.xx-5.1.3-x.x86_64
mmdsh -N NodeList rpm -e gpfs.docs-5.1.3-x.noarch
mmdsh -N NodeList rpm -e gpfs.gpl-5.1.3-x.noarch
mmdsh -N NodeList rpm -e gpfs.base-5.1.3-x.x86_64
      Ubuntu
      
mmdsh -N NodeList dpkg -P gpfs.base-debuginfo_5.1.3-x.x86_64
mmdsh -N NodeList dpkg -P gpfs.crypto_5.1.3-x.x86_64
mmdsh -N NodeList dpkg -P gpfs.adv_5.1.3-x.x86_64
mmdsh -N NodeList dpkg -P gpfs.msg.en-us_5.1.3-x.all
mmdsh -N NodeList dpkg -P gpfs.gskit_8.0.55.x.x86_64
mmdsh -N NodeList dpkg -P gpfs.compression_5.1.3-x.x86_64
mmdsh -N NodeList dpkg -P gpfs.license.xx_5.1.3-x.x86_64
mmdsh -N NodeList dpkg -P gpfs.docs_5.1.3-x.noarch
mmdsh -N NodeList dpkg -P gpfs.gpl_5.1.3-x.noarch
mmdsh -N NodeList dpkg -P gpfs.base_5.1.3-x.x86_64
      Note: In the preceding commands, the package name gpfs.license.xx needs to be changed depending on the IBM Spectrum Scale product edition.
  13. Reinstall GPFS.
  14. Proceed to Installation prerequisites and Using the installation toolkit to perform installation tasks: Explanations and examples.
Note: If you want to remove all cluster configurations, you can also apply the mmdelnode -f command to each node; however, if you choose to do so, you will also have to recreate cluster, BSDs, and file systems.