Cleanup procedures required if reinstalling with the installation toolkit

Before you can reinstall with the installation toolkit, you must perform some cleanup procedures first.

You can use the following procedures to clean up various stages of the previous GPFS™ installation and protocols deployment.

Starting the cleanup process

  1. Clean up the installer directory by issuing the following command:
    mmdsh -N NodeList rm -rf /usr/lpp/mmfs/4.2.3.0
    Note: The installer directory in this command depends upon the release version.
  2. Check /etc/resolv.conf for any changes that pointed the authentication directory to DNS, and remove them.
  3. If the object protocol was enabled and storage policies were created, use the mmobj policy list -v command and save the output list before you execute the mmces service disable OBJ command.

Cleaning up the authentication configurations

Use these commands to completely remove the current authentication configuration for all protocols, NFS/SMB only, or Object only. Be aware that doing so may cause loss of access to previously written data.

  1. Remove the configurations. To do so, issue the following commands:
    mmuserauth service remove --data-access-method file
mmuserauth service remove --data-access-method object
  2. Delete the ID mapping information when authentication was configured with AD. To do so, use the following command.
    Note: Once this ID mapping information is deleted, you might not be able to access existing files (or you might experience other unforeseen access issues), so do this only if you do not need access to existing data.
    mmuserauth service remove --data-access-method file --idmapdelete

Disabling CES

Disable CES on every node nodeNameX. To do so, issue the following command:
mmchnode -N nodeNameX --ces-disable

Protocol cleanup steps

  1. Use the following steps to clean up some of the object state on the system:
    mmdsh -N cesNodes systemctl stop postgresql-obj
rm -rf path_to_cesSharedRoot/object/keystone
    Delete the objectization temp directory. In the following example, the temp directory was created in filesystem fs1 at /ibm/fs1/ibmobjectizer/tmp:
    rm -rf  /ibm/fs1/ibmobjectizer/tmp
  2. Remove the fileset created for object.

    In the following example, the configured fileset object_fileset in file system fs1 was linked at /gpfs/fs1.

    mmlsfileset fs1
mmunlinkfileset fs1 object_fileset
mmdelfileset fs1 object_fileset -f
  3. Remove any fileset created for an object storage policy.

    Run the mmobj policy list -v command. If for some reason mmobj policy list -v cannot be executed (For example, if the Object Protocol or CES was already disabled) or you cannot detect which filesets are used for object storage policies, contact the IBM® Support Center.

    For example:

    If the mmobj policy list returned the following and the filesets got created in file system fs1:

    Index       Name          Deprecated  Fileset           Fileset Path                        Functions                Function Details
------------------------------------------------------------------------------------------------------------------------------------
0           SwiftDefault              object_fileset    /ibm/cesSharedRoot/object_fileset
11751509160 sof-policy1               obj_sof-policy1   /ibm/cesSharedRoot/obj_sof-policy1  file-and-object-access   regions="1"
11751509230 mysofpolicy               obj_mysofpolicy   /ibm/cesSharedRoot/obj_mysofpolicy  file-and-object-access   regions="1"
11751510260 Test19                    obj_Test19        /ibm/cesSharedRoot/obj_Test19                                regions="1"

    Use the following commands for each row except the first:

    mmunlinkfileset fs1 sof-policy1
mmdelfileset fs1 sof-policy1 -f
  4. Clean up installer remnants. To do so, issue the following command:
    mmdsh -N NodeList rm -rf /root/.chef
  5. Clear the cesSharedRoot configuration by issuing the following command:
    mmchconfig cesSharedRoot=DEFAULT
  6. Check your recent package installs to determine which packages need to be removed by issuing the following commands:
    RHEL 7.x and SLES 12
    rpm -qa --last|more
    Once you have determined which packages need to be removed, do so by issuing the following commands, as needed:
    RHEL 7.x and SLES 12
    mmdsh -N NodeList yum erase gpfs.smb nfs-ganesha-mount \
nfs-ganesha-vfs PyQt4 nfs-ganesha-utils phonon-backend-gstreamer phonon sip qt-x11 kde-filesystem \
qt qt-settings libmng nfs-ganesha-proxy nfs-ganesha-nullfs nfs-ganesha-gpfs nfs-ganesha chef -y

mmdsh -N NodeList yum erase python-webob MySQL-python mariadb-server perl-DBD-MySQL mariadb \
python-futures python-warlock python-jsonpatch python-jsonschema python-jsonpointer python-cmd2 \
python-cliff pyparsing memcached python-oslo-utils python-oslo-serialization python-oslo-i18n \
python-babel babel python-keyring python-stevedore python-prettytable python-pbr python-oslo-config \
python-netaddr python-iso8601 python-dnspython python-urllib3 python-six python-requests python-paste-deploy \
python-tempita python-paste python-netifaces python-simplejson python-greenlet python-eventlet -y

mmdsh -N NodeList yum erase gpfs.smb gpfs.smb-debuginfo sssd-ipa sssd python-sssdconfig sssd-proxy \
sssd-ldap sssd-krb5 libipa_hbac sssd-krb5-common sssd-common-pac sssd-ad sssd-common cyrus-sasl-gssapi c-ares \
libsss_idmap libdhash yp-tools ypbind -y

mmdsh -N NodeList yum erase python-routes python-repoze-lru python-oauthlib python-crypto \
python-qpid-common python-qpid python-dogpile-cache python-fixtures python-dogpile-core python-testtools \
python-extras python-oslo-context python-kombu python-anyjson python-amqp python-mimeparse \
python-markupsafe PyYAML python-passlib libyaml python-ibm-db-sa python-ibm-db python-sqlparse \
python-memcached python-webob python-posix_ipc python-sqlalchemy python-pbr python-greenlet -y

mmdsh -N NodeList yum erase nfs-ganesha gpfs.gss.pmsensors gpfs.gss.pmcollector gpfs.pm-ganesha boost-regex -y

mmdsh -N NodeList yum erase gpfs.smb python-oslo-i18n openstack-utils qt-settings nfs-ganesha-gpfs nfs-ganesha -y

mmdsh -N NodeList yum erase python-ordereddict python-routes python-passlib python-amqp python-crypto -y

mmdsh -N NodeList yum erase mariadb-libs -y

mmdsh -N NodeList yum erase pmswift nfs-ganesha-gpfs nfs-ganesha gpfs.smb -y

mmdsh -N NodeList yum erase python-cryptography python-enum34 swift3 python-pyeclib liberasurecode \
openstack-utils crudini python-msgpack python-wsgiref python-extras python-pycparser python-ply python-cffi \
python-ibm-db openstack-selinux xmlsec1 python-pyasn1 python-mimeparse python-retrying \
postgresql-server postgresql python-psycopg2 python-repoze-who jerasure gf-complete python-zope-interface \
postgresql-libs pytz python-oslo-context python-webob python-six nfs-ganesha-gpfs nfs-ganesha -y

mmdsh -N NodeList yum erase spectrum-scale-object spectrum-scale-object-selinux -y
  7. Issue rpm -qa --last|more on all nodes again, as the preceding list does not contain everything in the new builds.

    If the pmswift package still exists, remove it as follows:

    RHEL 7.x and SLES 12
    mmdsh -N NodeList rpm -e pmswift-4.2.3-0.noarch —noscripts

    You might also need to remove gpfs.smb.

  8. Clean up additional chef files. To do so, issue the following commands:
    mmdsh -N NodeList rm -rf /var/chef
mmdsh -N NodeList rm -rf /etc/chef
mmdsh -N NodeList rm -rf /opt/chef
mmdsh -N NodeList rm -rf /root/.chef
mmdsh -N NodeList rm -rf /root/.berkshelf
  9. Clean up performance monitoring tool files. To do so, issue the following commands:
    mmdsh -N NodeList rm -rf /opt/IBM/zimon*
mmdsh -N NodeList rm -rf /usr/IBM/zimon*
mmdsh -N NodeList rm -rf /var/log/cnlog/zimon*
mmdsh -N NodeList rm -rf /var/lib/yum/repose/x86_64/7Server/*zimon*
mmdsh -N NodeList rm -rf /var/lib/yum/repose/ppc64/7Server/*zimon*
  10. Clean up CTDB. To do so, issue the following commands:
    mmdsh -N NodeList rm -rf /var/lib/ctdb
  11. Clean up Swift files. To do so, issue the following commands:
    mmdsh -N NodeList rm -rf /etc/swift
mmdsh -N NodeList rm -rf /var/lib/mysql
mmdsh -N NodeList rm -rf /etc/keystone
mmdsh -N NodeList rm -rf /var/lib/keystone
mmdsh -N NodeList rm -rf /root/openrc
mmdsh -N NodeList rm -rf /var/cache/yum/x86_64/7Server/*
mmdsh -N NodeList rm -rf /var/cache/yum/ppc64/7Server/*
mmdsh -N NodeList rm -rf /var/log/maria*
mmdsh -N NodeList rm -rf /usr/share/pixmaps/comps/maria*
mmdsh -N NodeList rm -rf /var/log/keystone
mmdsh -N NodeList rm -rf /var/spool/cron/keystone
mmdsh -N NodeList rm -rf /usr/lib/python2.7/site-packages/sos/plugins/openstack_keystone*
mmdsh -N NodeList rm -rf /tmp/keystone-signing-swift
mmdsh -N NodeList rm -rf /usr/lib/python2.7/site-packages/sos/plugins/*
mmdsh -N NodeList rm -rf /var/lib/yum/repos/x86_64/7Server/icm_openstack
mmdsh -N NodeList rm -f /usr/lib/python2.7/site-packages/swiftonfile-2.5.0_0-py2.7.egg*
mmdsh -N NodeList rm -f /usr/bin/*objectizer*.pyc
mmdsh -N NodeList rm -f /usr/bin/generate_dbmap.pyc
mmdsh -N NodeList systemctl disable openstack-keystone.service
mmdsh -N NodeList systemctl disable openstack-swift-container-updater.service
mmdsh -N NodeList systemctl disable openstack-swift-container-update
mmdsh -N NodeList systemctl disable openstack-swift-object-updater.service
mmdsh -N NodeList systemctl disable openstack-swift-container-auditor.service
mmdsh -N NodeList systemctl disable openstack-swift-container-replicator.service
mmdsh -N NodeList systemctl disable openstack-swift-container.service
mmdsh -N NodeList systemctl disable openstack-swift-object-replicator.service
mmdsh -N NodeList systemctl disable openstack-swift-object.service
mmdsh -N NodeList systemctl disable openstack-keystone.service
mmdsh -N NodeList systemctl disable openstack-swift-account-reaper.service
mmdsh -N NodeList systemctl disable openstack-swift-account-auditor.service
mmdsh -N NodeList systemctl disable openstack-swift-account-replicator.service
mmdsh -N NodeList systemctl disable openstack-swift-account.service
mmdsh -N NodeList systemctl disable openstack-swift-proxy.service
mmdsh -N NodeList systemctl disable openstack-swift-object-auditor.service
mmdsh -N NodeList systemctl disable openstack-swift-object-expirer.service
mmdsh -N NodeList systemctl disable openstack-swift-container-reconciler.service
mmdsh -N NodeList rm -rf
/var/lib/yum/yumdb/o/0b01eb65826df92befd8c161798cb842fa3c941e-openstack-utils-2015.1-201502031913.ibm.el7.7-noarch
  12. Reboot the nodes. To do so, issue the following command:
    mmdsh -N NodeList shutdown -r now
    If some nodes do not fully come up, do the following:
    1. Power off that node.
    2. Wait one minute, then power the node back on.
      Note: Some nodes might need to be powered off and on more than once.
  13. Clean up Yum on all nodes. To do so, issue the following commands:
    mmdsh -N NodeList yum clean all
mmdsh -N NodeList rm -rf /etc/yum.repos.d/gpfs.repo /etc/yum.repos.d/icm* /etc/yum.repos.d/ces.repo 
mmdsh -N NodeList rm -rf /etc/yum.repos.d/epel* /etc/yum.repos.d/rdo*
mmdsh -N NodeList rm -rf /var/lib/yum/repos/x86_64/7Server/*
mmdsh -N NodeList rm -rf /var/lib/yum/repos/ppc64/7Server/*
  14. Remove GPFS.
    Notes:
    • This step is not required if you know that GPFS has not changed between old and new builds.
    • This step is not required if you prefer to perform an upgrade of GPFS.
    • This step is not required if you would like to keep the base GPFS installation intact and merely rerun the protocols deployment.
    • (To permanently remove GPFS, see Steps to permanently uninstall GPFS and/or Protocols.)
    1. Check which GPFS packages are on each node by issuing the following command:
      RHEL 7.x and SLES 12
      mmdsh -N NodeList rpm -qa|grep gpfs
      The system displays output similar to the following: 
      gpfs.ext-4.2.3-0.x86_64
gpfs.msg.en_US-4.2.3-0.noarch
gpfs.gskit-8.0.50-75.x86_64
gpfs.license.xx-4.2.3-0.x86_64
gpfs.crypto-4.2.3-0.x86_64
gpfs.adv-4.2.3-0.x86_64
gpfs.docs-4.2.3-0.noarch
gpfs.base-4.2.3-04.2.3-0.x86_64
gpfs.gpl-4.2.3-0.noarch
    2. Before removing anything, make sure that GPFS is shut down on all nodes. To do so, issue the following command:
      mmshutdown -a
    3. Remove the packages by issuing the following commands in the order shown.
      Note: When you remove gpfs.base, you will lose mmdsh access.
      Therefore, be sure to remove gpfs.base last, as shown here.
      RHEL 7.x and SLES 12
      mmdsh -N NodeList rpm -e gpfs.base-debuginfo-4.2.3-0.x86_64
mmdsh -N NodeList rpm -e gpfs.crypto-4.2.3-0.x86_64
mmdsh -N NodeList rpm -e gpfs.adv-4.2.3-0.x86_64
mmdsh -N NodeList rpm -e gpfs.ext-4.2.3-0.x86_64
mmdsh -N NodeList rpm -e gpfs.msg.en_US-4.2.3-0.noarch
mmdsh -N NodeList rpm -e gpfs.gskit-8.0.50-75.x86_64

mmdsh -N NodeList rpm -e gpfs.license.xx-4.2.3-0.x86_64
mmdsh -N NodeList rpm -e gpfs.docs-4.2.3-0.noarch
mmdsh -N NodeList rpm -e gpfs.gpl-4.2.3-0.noarch
mmdsh -N NodeList rpm -e gpfs.callhome-4.2.3-0.000.el7.noarch
mmdsh -N NodeList rpm -e gpfs.base-4.2.3-0.x86_64
      Note: In the preceding commands, the package name gpfs.license.xx needs to be changed depending on the IBM Spectrum Scale™ product edition.
  15. Reinstall GPFS.
  16. Proceed to Installation prerequisites and Using the installation toolkit to perform installation tasks: Explanations and examples.
Note: If you wish to remove all cluster configurations, you can also apply the mmdelnode -f command to each node; however, if you choose to do so, you will also have to remake cluster/nsds/filesystems.
Parent topic: Steps to permanently uninstall GPFS and/or Protocols