HDP 2.6.4 to HDP 3.1.0.0

For migrating from HDP 2.6.4 to HDP 3.1, ensure that you first map out and download all the software prerequisites. You must plan a cluster maintenance window and prepare for cluster downtime during the upgrade.

1. As the root user, download the management pack (as stated in Table 1) onto a directory on the Ambari server node. Ensure that the management pack is at a higher PTF version than the version of IBM Spectrum Scale service installed on your system. For information on downloading the management packs, see the topic IBM Spectrum Scale service (Mpack).
   Note: The downloaded management pack should be stored and unzipped in a different directory than the currently installed version of the Mpack.

   In this example, the downloaded management pack has been downloaded in the /root/GPFS_Ambari/upgrade_Mpack directory. The management pack contains the upgrade script to upgrade the Mpack.

   For example, if the currently installed Mpack is at 2.4.2.7 version, plan to upgrade to Mpack 2.7.0.3 version.

   The SpectrumScale_UpgradeIntegrationPackage script used for upgrade and migration is run from the /root/GPFS_Ambari/upgrade_Mpack directory.

   Ensure that the current Mpack installable package resides on a separate directory on the Ambari server node. This example uses the /root/GPFS_Ambari/currently_installed_Mpack directory.

   The SpectrumScaleMPackUninstaller.py script used as part of this procedure would have to be run from the /root/GPFS_Ambari/currently_installed_Mpack directory.

2. Log in to Ambari.
3. Disable short circuit if enabled.

   For more information, see Short-circuit read (SSR).

4. Generate an IBM Spectrum Scale snapshot.

   To create a snapshot, ensure that all POSIX and HDFS application and directory/file accesses are stopped.

   Ensure that IBM Spectrum Scale is active.

   If you are using shared file system via remote mount, execute the snapshot command on the Owning cluster.

   Check if /gpfs.mnt.dir/gpfs.data.dir is an independent fileset.

   Run mmlsfileset <filesystem> -L to check the InodeSpace value. If the InodeSpace is 0, then this is the root fileset. If the InodeSpace is a unique number, then this is an independent fileset.

   If this is an independent fileset, create the snapshot using the following command:

   mmcrsnapshot fsname snapshotname -j filesetname

   If this is not an independent fileset or if the gpfs.data.dir value is blank, then create a global file system snapshot using the following command:

   mmcrsnapshot fsname snapshotname

5. Stop all the services. Click Ambari > Actions > Stop All¹.

   ¹ - For FPO cluster, do not run STOP ALL from the Ambari GUI. Refer to the General section on how to properly stop IBM Spectrum Scale.

6. After all the services have stopped, unintegrate the transparency.
   Follow the steps in Unintegrating Transparency and ensure that the ambari-server restart command is run.

   Note: Do not start the services.
7. If the IBM Spectrum Scale service is not already stopped, click Ambari > Spectrum Scale > Service Actions > Stop.
8. On the Ambari server node as root, from the /root/GPFS_Ambari/upgrade_Mpack directory, run the SpectrumScale_UpgradeIntegrationPackage script with the --preEU option.

   The --preEU option saves the existing IBM Spectrum Scale service information into JSON files in the local directory where the script was run. It also removes the IBM Spectrum Scale service from the Ambari cluster so that the cluster can be properly migrated. This does not affect the IBM Spectrum Scale file system.

   Before you proceed, review the following questions for the upgrade script and have the information for your environment handy. If Kerberos is enabled, more inputs are required:

   Where the upgradeMpack=mpack2703
   
   [root@c902f10x09 mpack2703]# ./SpectrumScale_UpgradeIntegrationPackage --preEU
   Are you sure you want to upgrade the GPFS Ambari integration package (Y/N)? (Default Y):
   ************************************************************
   ***STARTING WITH PRE EXPRESS UPGRADE STEPS***
   ************************************************************
   Enter the Ambari server username:(Default admin ):
   Enter the password for the Ambari server.
   Password:
   Retype password:
   SSL Enabled (True/False) (Default False):
   Enter the Ambari server Port. (Default 8080):
   http://c902f10x09.gpfs.net:8080
   {
     "href" : "http://c902f10x09.gpfs.net:8080/api/v1/clusters",
   …
   Service STATEtrue
   Successfully completed DELETE call to remove the Spectrum Scale service.
   …
   Starting ambari-server
   Ambari Server running with administrator privileges.
   Organizing resource files at /var/lib/ambari-server/resources...
   Ambari database consistency check started...
   No errors were found.
   Ambari database consistency check finished
   Server PID at: /var/run/ambari-server/ambari-server.pid
   Server out at: /var/log/ambari-server/ambari-server.out
   Server log at: /var/log/ambari-server/ambari-server.log
   Waiting for server start....................
   Ambari Server 'start' completed successfully.
   [root@c902f10x09 upgradeMpack]#
   

9. As a root user on the Ambari server, run the Mpack uninstaller script, SpectrumScaleMPackUninstaller.py, from the currently installed Mpack directory, to remove the existing Mpack link in Ambari.

   Where the currently_installed_Mpack=mpack2427
   
   [root@c902f10x09 mpack2427]# ./SpectrumScaleMPackUninstaller.py
   INFO: ***Starting the Mpack Uninstaller***   
   
   Enter Ambari Server Port Number. If it is not entered, the uninstaller will take default port 8080  :   
   INFO: Taking default port 8080 as Ambari Server Port Number.
   Enter Ambari Server IP Address  :   192.0.2.22
   Enter Ambari Server Username, default=admin :   
   INFO: Taking default username "admin" as Ambari Server Username.
   Enter Ambari Server Password  :   
   INFO: Verifying Ambari Server Address, Username and Password.
   INFO: Verification Successful.
   INFO: Spectrum Scale Service is not added to Ambari.
   INFO: Spectrum Scale MPack Exists. Removing the MPack.
   INFO: Reverting back Spectrum Scale Changes performed while mpack installation.
   INFO: Deleted the Spectrum Scale Link Successfully.
   INFO: Removing Spectrum Scale MPack.
   INFO: Performing Ambari Server Restart.
   INFO: Ambari Server Restart Completed Successfully.
   INFO: Spectrum Scale Mpack Removal Successfully Completed.
   [root@c902f10x09 mpack2420]#
   
   

10. Start all services. Click Ambari > Actions > Start All.

    Wait for all the services to start. At this stage, native HDFS is used.

    Check to ensure that the HDFS Transparency is not active, by executing the following commands:

    • On NameNodes: ps -eaf | grep namenode | grep -v mmfs
    • On DataNodes: ps -eaf | grep datanode | grep -v mmfs

    Now, HDP is in the native HDFS mode.

11. To upgrade from HDP 2.6.4 to HDP3.1, refer to the Hortonworks migration guide for the following procedures depending on the specific architecture:

    Upgrading to HDP 3.1 on Power®

    Upgrading to HDP 3.1 for x86_64

    Note:

    • When migrating to HDP in an x86 environment, ensure that the procedure given in the Switch from IBM® Open JDK to Oracle JDK section is completed.
    • Ensure that you properly follow the Hortonworks HDP migration guide. Some steps to take extra notices on:
      • In “Preparing to Upgrade Ambari”:
        • Put the ambari-metrics into maintenance mode.
        • Make a safe copy of the Ambari server configuration file (/etc/ambari-server/conf/ambari.properties.3)
      • Ensure that all the services are up and active, all the critical alerts are resolved, and all the service check passed before performing the express upgrade to HDP 2.6.4.
      • In “Upgrade Ambari":

        If you are using the default Postgres database for Ambari server, you need to upgrade Postgres to a supported version. For more information, see Hortonworks documentation.

        Back-up your existing Ambari database before upgrading the Ambari server database. For example, HDP 3.1.0 requires Postgres 9.6 or 10.2.

        Postgres must be upgraded before the Ambari server is upgraded.

        On Power systems, Postgres has dependencies on the advance-toolchain-at*-runtime, advance-toolchain-at*-devel, and advance-toolchain-at*-perf packages. Install the advance-toolchain before upgrading Postgres.

        Note: Remove the current Postgres version and re-install with a new Postgres version on the Power systems to avoid the following error:

        "Checking cluster versions /usr/bin/pg_ctl-orig: relocation error: /opt/at10.0/lib64/power8/libpthread.so.0: symbol __libc_vfork, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference could not get pg_ctl version data using “/usr/bin/pg_ctl” --version: No such file or directory."

        This is due to dependencies issues with the advance-toolchain on Power systems.

      • In “Post-upgrade Tasks”:

        Hive Post-upgrade Tasks: If the data directory resides in IBM Spectrum Scale, then the Hive directory changes would need to be run when the IBM Spectrum Scale service is re-integrated.

12. After Ambari and HDP are upgraded, ensure that you stop all the services before you proceed to re-deploy the IBM Spectrum Scale service.

    Click Ambari > Actions > Stop All.

    Wait until all services have stopped. Ensure that the native HDFS has stopped running.

13. HDP 3.1.x supports HDFS Transparency version 3.1.0-x and later. Only HDFS Transparency 3.1.0 stream is supported by HDP.

    Add the HDFS Transparency version as stated in Table 1 into the GPFS repo directory.

    Ensure that the older HDFS Transparency version is removed from the repo directory because only one HDFS Transparency rpm can reside in the GPFS repo directory.

    Run "createrepo . " to update the repo metadata.

14. Add the IBM Spectrum Scale service.

    On the Ambari server node as root, from the /root/GPFS_Ambari/upgrade_Mpack directory, run the SpectrumScale_UpgradeIntegrationPackage script with the --postEU option in the directory where the --preEU step was run and where the JSON configurations were stored.

    Before you proceed, for the --postEU option, review the following questions, and have the information for your environment handy. If Kerberos is enabled, more inputs are required.

    Where the upgradeMpack=mpack2703
    
    [root@c902f10x09 mpack2703]# ./SpectrumScale_UpgradeIntegrationPackage --postEU
    Are you sure you want to upgrade the GPFS Ambari integration package (Y/N)? (Default Y):
    *************************************************************
    ***STARTING WITH SPECTRUM SCALE EXPRESS UPGRADE POST STEPS***
    *************************************************************
    Starting Post Express Upgrade Steps. Enter Credentials
    Enter the Ambari server User:(Default admin ):
    Enter the password for the Ambari server.
    Password:
    Retype password:
    SSL Enabled (True/False) (Default False):
    Enter the Ambari server Port. (Default 8080):
    ....
    # Accept License
    Do you agree to the above license terms? [yes or no]
    yes
    Installing...
    Enter Ambari Server Port Number. If it is not entered, the installer will take default port 8080:
    INFO: Taking default port 8080 as Ambari Server Port Number.
    Enter Ambari Server IP Address : 192.0.2.22
    Enter Ambari Server Username, default=admin :
    INFO: Taking default username "admin" as Ambari Server Username.
    Enter Ambari Server Password :
    ...
    Enter kdc principal:
    Enter kdc password:
    ...
    From the Ambari GUI, check the IBM Spectrum Scale installation progress through the background
    operations panel.
    Enter Y only when installation of the Spectrum Scale service using REST call process is completed.
    (Default N)Y ** SEE NOTE BELOW **
    Waiting for the Spectrum Scale service to be completely installed.
    ...
    Waiting for server start....................
    Ambari Server ’start’ completed successfully.
    *************************************************************
    Upgrade of the Spectrum Scale Service completed successfully.
    *************************************************************
    ***********************************************************************************************
    IMPORTANT: You need to ensure that the HDFS Transparency package, gpfs.hdfs-protocol-3.0.x,
    is updated in the Spectrum Scale repository. Then follow the "Upgrade Transparency" service
    action in the Spectrum Scale service UI panel to propagate the package to all the GPFS Nodes.
    After that is completed, invoke the "Start All" services in Ambari.
    ***********************************************************************************************
    
    

15. Update the HDFS Transparency package to all the GPFS nodes.

    HDP 3.1.x requires HDFS Transparency version 3.1.0-x. Update the HDFS Transparency package before you start any services.

    Ensure that the HDFS Transparency package, gpfs.hdfs-protocol-3.1.0.X, is updated in the IBM Spectrum Scale repository as stated in Step 11.

    From Ambari GUI, go to Upgrade Transparency service action in the Spectrum Scale service UI window to propagate the new package to all the GPFS Nodes. For more information, see Upgrading Transparency.

    Ensure to check that all the GPFS nodes have the HDFS Transparency upgraded to the correct version by running the following command:

    mmdsh -N all “rpm -qa | grep gpfs.hdfs-protocol”

16. Start all services.

    Click Ambari > Actions > Start All.

    Restart all components by using the Restart icon.

17. If short circuit was disabled earlier and needs to be enabled, enable it now.

    See Short-circuit read (SSR).

    Note:

    • If the IBM Spectrum Scale service is restarted by using the restart icon, the HDFS service also needs to be restarted.
    • The NameNode Last Checkpoint alert, NameNode Blocks Health alert and NameNode HDFS Pending Deletion Blocks alert can be ignored and disabled.
    • If the HBase master failed to start with FileAlreadyExistsException error, restart HDFS and then restart the HBase master.