Upgrading HDP 3.1.x non-HA

This section describes the HDP and IBM Spectrum® Scale MPack upgrade process for HDP 3.1.x non-HA and HDP 3.0.x and earlier.

In the Upgrading HDP overview section, see Figure 2 for the flow.

You must plan a cluster maintenance window and prepare for cluster downtime when you upgrade the IBM Spectrum Scale MPack.

Note:
  • You must perform the Mpack upgrade only if the target Mpack version is supported on your HDP level. Ensure that you check the support matrix and verify whether the Mpack version is supported with your HDP level.
  • To see the default configuration modifications under IBM Spectrum Scale Mpacks, refer to the Big data and analytics section under the .
  • For HDP upgrade, only express upgrade is supported.
  • The cluster must be at management pack version 2.7.0.0 or later.
  • Upgrading MPack does not affect the IBM Spectrum Scale file system.
  • Ensure that the anonymous user id is created and have the same uid/gid in your cluster before upgrading. From Mpack 2.4.2.6, having an anonymous user id is mandatory. For more information, see Create the anonymous user id.
  • Before you proceed with the upgrade process in a Kerberized environment, you need to set the KDC_PRINCIPAL and KDC_PRINCIPAL_PASSWORD values in the IBM Spectrum Scale services > Configs > Advanced section and save the configuration. If the environment is Kerberized, the unintegrate HDFS Transparency service action requires the KDC_PRINCIPAL and KDC_PRINCIPAL_PASSWORD values to be configured in advance.
  • If you are planning to migrate from a Mpack version 2.7.0.3 or earlier to Mpack version 2.7.0.4 or later, a workaround solution is required. For information, see Upgrade failures from Mpack 2.7.0.3 or earlier to Mpack 2.7.0.4 - 2.7.0.6.
Procedure
  1. As the root user, download a management pack at a higher PTF version than the version of IBM Spectrum Scale service installed on your system, onto a directory on the Ambari server node. For information on downloading the management packs, see IBM Spectrum Scale service (Mpack).
    Note: The downloaded management pack should be stored and unzipped in a directory different than the currently installed version of the Mpack.

    In this example, the downloaded management pack has been downloaded in the /root/GPFS_Ambari/upgrade_Mpack directory. The management pack contains the upgrade script to upgrade the MPack.

    For example, if the currently installed Mpack is at 2.7.0.0 version then plan to upgrade to Mpack 2.7.0.1 version.

    The SpectrumScale_UpgradeIntegrationPackage script used for upgrade and migration is run from the /root/GPFS_Ambari/upgrade_Mpack directory.

    Ensure that the current Mpack installable package resides on a separate directory on the Ambari server node. This example uses the /root/GPFS_Ambari/currently_installed_Mpack directory.

    The SpectrumScaleMPackUninstaller.py script used as part of this procedure would have to be run from the /root/GPFS_Ambari/currently_installed_Mpack directory.

  2. Log in to Ambari.
  3. Stop all the services. Click Ambari > Actions > Stop All1.
    1For FPO cluster, do not run STOP ALL from the Ambari GUI. Refer to the Limitations > General sections on how to properly stop IBM Spectrum Scale.
    Note: Ensure that the IBM Spectrum Scale file system is not being accessed using either HDFS or POSIX so that it can be unmounted and stopped properly. For more information, see Why did the IBM Spectrum Scale service did not stop or restart properly? in the General Problem determination section.
  4. After all the services have stopped, unintegrate the transparency.
    Note: If you run the unintegrate HDFS Transparency more than once consecutively, unpredictable errors will occur and would cause the cluster to be in an unusable state. In such cases, contact scale@us.ibm.com.
    To unintegrate the transparency, run the following steps:
    1. Click Spectrum Scale > Service Actions > Unintegrate Transparency.
    2. On the Ambari server node, run the ambari-server restart command to restart the Ambari server.
      Note: Do not start any services.
  5. If the IBM Spectrum Scale service is not already stopped, stop the IBM Spectrum Scale service by clicking Ambari > Spectrum Scale > Service Actions > Stop.
  6. As the root user on the Ambari server node, from the /root/GPFS_Ambari/upgrade_Mpack directory, run the SpectrumScale_UpgradeIntegrationPackage script with the preEU option.

    The --preEU option saves the existing IBM Spectrum Scale service information into JSON files in the local directory where the script was run. It also removes the IBM Spectrum Scale service from the Ambari cluster so that the BI cluster can be properly migrated. This does not affect the IBM Spectrum Scale file system.

    Note: If you are migrating from Mpack version 2.7.0.3 or earlier to Mpack version 2.7.0.4 or later, run the SpectrumScale_UpgradeIntegrationPackage script with the preEU option command from the currently_installed_Mpack instead. Then copy the generated files specified in Upgrade failures from Mpack 2.7.0.3 or earlier to Mpack 2.7.0.4 - 2.7.0.6. to the upgrade_Mpack directory.

    Before you proceed, review the following questions for the upgrade script and have the information for your environment handy. If Kerberos is enabled, more inputs are required.

    $ cd /root/GPFS_Ambari/upgrade_Mpack
$ ./SpectrumScale_UpgradeIntegrationPackage --preEU
Are you sure you want to upgrade the GPFS Ambari integration package (Y/N)? (Default Y):
************************************************************
***STARTING WITH SPECTRUM SCALE EXPRESS UPGRADE PRE STEPS***
************************************************************
Enter the Ambari server User:(Default admin ):
Enter the password for the Ambari server.
Password:
Retype password:
SSL Enabled (True/False) (Default False):
Enter the Ambari server Port. (Default 8080):
...
# Note: If Kerberos is enabled, then the KDC principal and password information are required.
Kerberos is Enabled. Proceeding with Configuration
Enter kdc principal:
Enter kdc password:
...
  7. As a root user on the Ambari server, run the MPack uninstaller script, SpectrumScaleMPackUninstaller.py, from the currently installed Mpack directory, to remove the existing MPack link in Ambari.

    The removal of the IBM Spectrum Scale service during the SpectrumScale_UpgradeIntegrationPackage --preEU does not remove the Mpack link in the Ambari database. After the service is removed, remove the link.

    
$ cd /root/GPFS_Ambari/currently_installed_Mpack
$./SpectrumScaleMPackUninstaller.py 
INFO: ***Starting the MPack Uninstaller*** 

Enter Ambari Server Port Number. If it is not entered, the uninstaller will take default port 8080: 
INFO: Taking default port 8080 as Ambari Server Port Number.
Enter Ambari Server IP Address : 192.0.2.22
Enter Ambari Server Username, default=admin : 
INFO: Taking default username "admin" as Ambari Server Username.
Enter Ambari Server Password : 
INFO: Verifying Ambari Server Address, Username and Password.
INFO: Verification Successful.
INFO: Spectrum Scale Service is not added to Ambari.
INFO: Spectrum Scale MPack Exists. Removing the MPack.
INFO: Reverting back Spectrum Scale Changes performed while MPack installation.
INFO: Deleted the Spectrum Scale Link Successfully.
INFO: Removing Spectrum Scale MPack.
INFO: Performing Ambari Server Restart.
INFO: Ambari Server Restart Completed Successfully.
INFO: Spectrum Scale MPack Removal Successfully Completed.
  8. After you are in native HDFS, log in to the Ambari server and perform the following checks:
    1. Check the Directories section under Customize services to ensure that the service field names values do not contain any IBM Spectrum Scale directory paths. If there are any, remove those paths and save the configuration. For example, check values for the following fields:
      dfs.datanode.data.dir 
dfs.namenode.name.dir
yarn.nodemanager.log-dirs
yarn.nodemanager.local-dirs
    2. IBM Spectrum Scale service does not have journal nodes. However, after it is back in native HDFS, the journal nodes are restored. If you are using Kerberos, the journal nodes are required to have the proper principals configured. If not, you need to create the principals for them after unintegrating HDFS transparency.
  9. HDP is now in the native HDFS mode.
    • If you plan to upgrade HDP to a newer level, follow the process defined in the Hortonworks documentation to upgrade the HDP and the Ambari versions that the Mpack level supports.
    • After HDP and Ambari are upgraded, ensure that you stop all the services before you proceed to re-deploy the IBM Spectrum Scale service.
  10. Ensure all services have stopped.
  11. On the Ambari server node as root, from the /root/GPFS_Ambari/upgrade_Mpack directory, run the SpectrumScale_UpgradeIntegrationPackage script with the --postEU option in the directory where the --preEU step was run and where the JSON configurations were stored.
    Note: If you are migrating from Mpack version 2.7.0.3 or earlier to Mpack version 2.7.0.4 or later, ensure that the generated files specified in Upgrade failures from Mpack 2.7.0.3 or earlier to Mpack 2.7.0.4 - 2.7.0.6. are copied to the upgrade_Mpack directory before running the SpectrumScale_UpgradeIntegrationPackage script --postEU command.
    Before you proceed, for the --postEU option, review the following questions and have the information for your environment handy. If Kerberos is enabled, more inputs are required.
    
$ cd /root/GPFS_Ambari/upgrade_Mpack
$ ./SpectrumScale_UpgradeIntegrationPackage --postEU
Are you sure you want to upgrade the GPFS Ambari integration package (Y/N)? (Default Y):
*************************************************************
***STARTING WITH SPECTRUM SCALE EXPRESS UPGRADE POST STEPS***
*************************************************************
Starting Post Express Upgrade Steps. Enter Credentials
Enter the Ambari server User:(Default admin ):
Enter the password for the Ambari server.
Password:
Retype password:
SSL Enabled (True/False) (Default False):
Enter the Ambari server Port. (Default 8080):
....
# Accept License
Do you agree to the above license terms? [yes or no]
yes
Installing...
Enter Ambari Server Port Number. If it is not entered, the installer will take default port 8080 :
INFO: Taking default port 8080 as Ambari Server Port Number.
Enter Ambari Server IP Address :
192.0.2.22
Enter Ambari Server Username, default=admin :
INFO: Taking default username "admin" as Ambari Server Username.
Enter Ambari Server Password :
...
Enter kdc principal:
Enter kdc password:
...
From the Ambari GUI, check the IBM Spectrum Scale installation progress through the background
operations panel.
Enter Y only when installation of the Spectrum Scale service using REST call process is completed.
(Default N)Y ** SEE NOTE BELOW **
Waiting for the Spectrum Scale service to be completely installed.
...
Waiting for server start....................
Ambari Server ’start’ completed successfully.
*************************************************************
Upgrade of the Spectrum Scale Service completed successfully.
*************************************************************
***********************************************************************************************
IMPORTANT: You need to ensure that the HDFS Transparency package, gpfs.hdfs-protocol-2.7.3.X,
is updated in the Spectrum Scale repository. Then follow the "Upgrade Transparency" service
action in the Spectrum Scale service UI panel to propagate the package to all the GPFS Nodes.
After that is completed, invoke the "Start All" services in Ambari.
***********************************************************************************************
    Note: If the Mpack requires a corresponding HDFS Transparency update version, ensure that the process in the Upgrading HDFS Transparency is done before doing a Start All in the next step.
  12. Start all the services.

    Click Ambari > Actions > Start All.

    Restart all the components by using the restart icon.

    Note:
    • If the Start All fails, try starting each of the services individually. Ensure that the manual starting services in the Ambari order is executed first. For more information, see Manually starting services in Ambari.
    • If the IBM Spectrum Scale service is restarted by using the restart icon, the HDFS service also needs to be restarted.
    • The NameNode Last Checkpoint alert can be ignored and can be disabled.
    • If the HBase master failed to start with FileAlreadyExistsException error, restart HDFS and then restart the HBase master.