Patch management process for BigInsights Big SQL 4.2

Answer

Requirements
The following information will be required to run the patch management process:

• The protocol (http/https), host name & port number of the Apache Ambari server for the cluster
• The Apache Ambari administrator user name and password
• If the cluster has Kerberos security enabled, the Kerberos administrator credentials (principal & password)

In addition, to upgrade Big SQL, there must exist a user with the following attributes:

• Passwordless sudo access on all nodes of the cluster, including the Ambari server itself.
• The ability to connect passwordlessly through ssh from the Ambari server to all Big SQL nodes.

This user can be root. If the user is not root, the user name must be passed to the upgrade script with the -a option. The upgrade script must be run with root user privilege. This can be achieved by using the sudo command.

The current patch management process requires about 4 Gb of free disk space on the root (/) partition. This space is temporarily used by the installation process and released once the installation is complete. If this space is not available, the patch management prerequisite checker will warn you and stop the installation process.

Apache Ambari configuration groups are not supported by the patch management process. If configuration groups are present, the patch management prerequisite checker will warn you and stop the installation process. It is recommended to disable / remove configuration groups before initiating the patch management process.

The HDFS service must be up and running in the Apache Ambari server. If it is not running, the patch management prerequisite checker will warn you and stop the installation process.

The Big SQL High Availability (HA) feature must be disabled on the cluster, and there must be only one head node for the installation to proceed. If HA is enabled, the patch management prerequisite checker will warn you and stop the installation process.

Overview of the Fix Deployment Process
The patch installation process is a manual, scripted process which consists of three distinct phases that must be run in sequence by executing the bigsql_upgrade.py script with the appropriate options.
The three phases are Backup, Upgrade and Finalize.

During the upgrade process, the state of the BigInsights - Big SQL service might be reported incorrectly in the Apache Ambari server web interface. This is expected behaviour. It will return to normal after the upgrade is complete.

The script to install the patch is located at
/usr/ibmpacks/bin/<version>/upgrade/bigsql_upgrade.py on the Ambari server host after the BigInsights value-add package is deployed on the system.

The script accepts the following options:
-m phase
Required - The upgrade phase to run. Valid values are: Backup, Upgrade, Finalize, or Restore.
-u ambari admin user
Optional - The Ambari admin user name. This default is admin.
-p ambari admin pwd
Optional - Ambari admin password. This defaults to admin.
-s http protocol
Optional - Ambari server protocol ('http' or 'https') . This defaults to http.
-P ambari server port
Optional - Ambari server port number. This defaults to 8080.
-k kerberos admin principal
Optional - Kerberos admin principal. This is required if Kerberos is enabled on the cluster.
-w kerberos admin password
Optional - Kerberos admin password. This is required if Kerberos is enabled on the cluster.
-a ssh user
Optional - Identity used for inter-node passwordless ssh connections. This defaults to root.

Example invocation of the script

• The Ambari dashboard is ambari.company.com:9443
• The administrator user name is 'ambadmin' and the password is 'ambpassword'.
• Kerberos is not enabled.
• The environment is set up for password less ssh connectivity between nodes for a user named 'ssh_user'. This user also has password less sudo rights.
• You are logged in as the root user.

To run the 'Backup' phase of the Big SQL service patch management process on your BigInsights cluster, invoke the script in the following way
/usr/ibmpacks/bin/<version>/upgrade/bigsql_upgrade.py -u ambadmin -p ambpassword -s https -P 9443 -a ssh_user -m Backup

Script Logging & Troubleshooting
A complete log of the script execution is located at /var/ibm/bigsql/logs/upgrade.log on the Ambari server node (the one where the script is executed).
Logs for additional subtasks performed by the script on individual nodes of the cluster will be located in the same directory on the individual cluster (head and worker) nodes.
Additional logs related to the upgrade of the Big SQL engine will be found in the /tmp and /tmp/bigsql/logs directory on the Big SQL head node.

Fix Installation Instructions
Obtaining the Fix
When a defect is reported and fixed, a new BigInsights Value-Add build containing the fix will be generated and provided to you as a TAR archive by IBM Technical Support. This archive must be used to set up a new repo site in a network location accessible by all the nodes in the BigInsights cluster. This process is very similar to performing an OFFLINE installation of BigInsights and is detailed below.

The patch management process expects two repos: the current repo with the code level in effect, and a new one with the patch being provided.

Setting up a Local Repo Site
The following steps will walk you through the process of creating a local repository to make the fix installation files available to the nodes in your cluster. This must be done on a network node that can be accessed by all nodes in the BigInsights cluster. If you do not have an additional Linux host, you can use one of the Hadoop management nodes. The repository server requires an HTTP web server. The steps below explains how to install it.

1. On the repository server, if the Apache HTTP server is not installed, install it:
sudo yum install httpd

2. On the repository server, ensure that the createrepo package is installed.
sudo yum install createrepo

3. Make sure there is network access from all nodes in your cluster to the repository server. If data nodes are on a private network and the repository server is external to the Hadoop cluster, you might need to configure iptables for IP forwarding and masquerading.

4. On the repository server, create a new directory for your value-add repository, such as <mirror web server document root>/repos/patch. For example, for Apache httpd, the default is /var/www/html/repos.
You can use any name you like for the repository directory.
sudo mkdir /var/www/html/repos/patch

5. Expand the TAR archive received from IBM Technical Support into the mirror web server location, <your.mirror.web.server.document root>/repos/patch directory. Note that their exact names will depend on the platform & OS family you cluster is running on.
sudo tar -xvzf biginsights_4.2.0.0.el6.x86_64.tar.gz -C /var/www/html/repos/patch/

6. Start the web server. If you use Apache httpd, start it by using either of the following
commands:
sudo apachectl start
or
sudo service httpd start

7. Ensure that any firewall settings allow inbound HTTP access from your cluster nodes to the mirror web server.

8. Test your local repository by browsing to the web directory. Note that the exact path depends on the OS & platform your cluster is running on. Please substitute as appropriate.
http://<your.mirror.web.server>/repos/patch/BigInsights-Valuepacks/RHEL6/x86_64/4.2.0.0/

9. You should see all of the files that you copied to the repository server. On the repository server, run the createrepo command to initialize the repository, again substituting as appropriate.
createrepo /var/www/html/repos/patch/BigInsights-Valuepacks/RHEL6/x86_64/4.2.0.0/


Install the BigInsights Value-Add RPM

IMPORTANT NOTE:

• If you are on Big SQL Dec refresh build and applying a patch, please make a backup copy of /var/lib/ambari-server/resources/stacks/BigInsights/4.2/repos/repoinfo.xml before installing the BigInsights Value-Add RPM.

cp /var/lib/ambari-server/resources/stacks/BigInsights/4.2/repos/repoinfo.xml /var/lib/ambari-server/resources/stacks/BigInsights/4.2/repos/repoinfo.xml.bk

• Ensure there are no repoinfo xml files ending with .back (i.e repoinfo.xml-04_09_2018_09_59_57OURCE.back )

rm -rf *OURCE.back

How to verify you are on Dec Refresh BigInsights value add RPM:

yum info BigInsights-IOP-2.13.1-IOP-4_2 | grep Release
Release : SNAPSHOT20161215020000 << This timestamp points to Dec Refresh RPM


1). Locate the BigInsights Value-Add RPM to install on the Ambari Server host of the Hadoop cluster in the new repo you just created; for example, the /var/www/html/repos/patch/BigInsights-Valuepacks/RHEL6/x86_64/4.2.0.0/ directory –note, that the name may be different depending on the OS family & platform your cluster is running on
BigInsights-IOP-1.2.0.0-4.2.<family>.<platform>.rpm

2. Copy this RPM file to the Ambari Server host and install it by using the following commands:
sudo yum localinstall BigInsights-IOP-1.2.0.0-4.2.<family>.<platform>.rpm

3. On the Ambari Server node, navigate to the /var/lib/ambari-server/resources/stacks/BigInsights/4.2/repos/repoinfo.xml file. Ensure the <baseurl> element for the BIGINSIGHTS-VALUEPACK-1.2.0.0<repo> entry points to your repository server. Remember, there might be multiple <repo> sections. Make sure that the URL you tested in step 8 matches exactly the value indicated in the <baseurl> element. For example, the repoinfo.xml might look like the following after your changes are applied (the part in bold indicating the changes that you have performed):
...
<repo>
<baseurl> http://<your.mirror.web.server>/repos/patch/BigInsights-Valuepacks/RHEL6/x86_64/4.2.0.0 </baseurl>
<repoid>BIGINSIGHTS-VALUEPACK-1.2.0.0</repoid>
<reponame>BIGINSIGHTS-VALUEPACK-1.2.0.0</reponame>
</repo>
...

The new <repo> section might appear as a single line.

IMPORTANT NOTE:
If you are patching from Big SQL Dec Refresh, after installing BigInsights rpm from step #2, rename repoinfo.xml.bk (backed up file from step #1) to a new .back file with the latest timestamp like below:

mv /var/lib/ambari-server/resources/stacks/BigInsights/4.2/repos/repoinfo.xml.bk /var/lib/ambari-server/resources/stacks/BigInsights/4.2/repos/repoinfo.xml-$( date +'%m_%d_%Y_%H_%M_%S')OURCE.back

Note: Purpose of the above step is to ensure the .back file with the latest timestamp contains the repo information (baseurl) pointing to your December Refresh repository.

4. Clear the yum cache.
yum clean all

5. Restart the Ambari server.
ambari-server restart

6. Give the Ambari server a few minutes to fully replicate its updated configuration to all its nodes after the restart. On small clusters (less than 5 nodes) a minute after the server has restarted should be enough. On bigger cluster, it is recommended to wait about 10 minutes.

Running the Backup phase of the Fix Deployment Process
The “Backup” phase is responsible for backing up on all Big SQL nodes the Big SQL catalog,configuration information and data for the pure DB2 tables (ie tables that are not Hadoop or Hbase
tables). Please note that the data for Hadoop & Hbase tables reside on HDFS and is not touched by the fix deployment process.

To perform the Backup phase, execute the Big SQL upgrade script with the -m Backup option. Add any other options required by your environment as described in the “Overview of the Fix Deployment Process” section of this document.
python /usr/ibmpacks/bin/3.0-SNAPSHOT/upgrade/bigsql_upgrade.py -m Backup

The Backup phase will take about 15 minutes. This time may vary depending on the specs of the machines & the bandwidthof the link between the repo site and the cluster nodes. On successful completion of the Backup phase, the script will issue the following message:

INFO: #### Big SQL upgrade 'Backup' phase completed successfully ####

At the end of the Backup phase, the Big SQL catalog, configuration information and data for the pure DB2 tables are backed up. The Big SQL service is NOT RUNNING and DOES NOT show in the Ambari dashboard. This is the expected behavior. If needed, Big SQL service can be started
from the command line and used. Please note that in that case, the fix is not effective yet.

In case of failure reported by the script during the Backup phase, identify the cause of the issue (access control, lack of disk space, etc ...), resolve it and re-run the backup phase. The backup phase
is designed to be re-run as many times as needed.

Running the Upgrade phase of the Fix Deployment Process
The “Upgrade” phase is responsible for installing the Big SQL binaries containing the fix, and perform any additional management task required by the fix.

To perform the Upgrade phase, execute the Big SQL upgrade script with the -m Upgrade option. Add any other options required by your environment as described in the “Overview of the Fix
Deployment Process” section of this document.
python /usr/ibmpacks/bin/3.0-SNAPSHOT/upgrade/bigsql_upgrade.py -m Upgrade

The Upgrade phase will take about 25 minutes. This time may vary depending on the specs of the machines & the number of Big SQL objects (tables, etc ...).

At the end of the Upgrade phase, the new Big SQL binaries are deployed. The Big SQL service is RUNNING and DOES NOT show in the Ambari dashboard. This is the expected behaviour. The Big SQL service can be used to run some test, evaluate and validates from the command line or through an standard JDBC connection as the fix is now effective.

In case of failure reported by the script during the Upgrade phase, DO NOT re-run the Upgrade phase (the script will prevent it). Instead, identify the cause of the issue (access control, lack of disk space, etc ...), resolve it and run the Restore phase as explained in the “Running the Restore phase
of the Fix Deployment Process” section of this document.

Running the Finalize phase of the Fix Deployment Process
IMPORTANT NOTE: Once the Finalize phase has run, the fix installation is definitive and cannot be reverted. Back-ups created during the backup phase are also cleaned up. It is recommended to verify that the installation of the fix has been successfull by running test, evaluate and validates before running the
Finalize phase.

The “Finalize” phase is responsible for finalizig the installation and cleaning up backups and other work files created during the fix deployment process.

To perform the Finalize phase, execute the Big SQL upgrade script with the -m Finalize option. Add any other options required by your environment as described in the “Overview of the Fix Deployment Process” section of this document.
python /usr/ibmpacks/bin/3.0-SNAPSHOT/upgrade/bigsql_upgrade.py -m Finalize

The Finalize phase will take about 3 minutes. This time may vary depending on the specs of the machines.

At the end of the Finalize phase, the Big SQL service is running and showing the Ambari dashboard. The fix is now deployed and the Big SQL service is ready to use again.
In case of failure reported by the script during the Finalize phase, identify the cause of the issue (access control, lack of disk space, etc ...), resolve it and re-run the Finalize phase. The Finalize phase is designed to be re-run as many times as needed.

Running the Restore phase of the Fix Deployment Process
IMPORTANT NOTE: This phase must only be run in case of failure (in the Upgrade phase) or to remove a deffective fix.

In case of failures of the Upgrade phase, it is necessary to restore the backup to bring back the cluster to a state where the Upgrade phase can be re-attempted. The Restore phase can also be used
to remove a fix that is proving deffective, assuming that the Finalize phase has not be run yet.

The “Restore” phase is responsible for restoring the backups taken during the Backup phase and bringing back the cluster to its initial state before the Backup phase was run.
To perform the Restore phase, execute the Big SQL upgrade script with the -m Restore option.

Add any other options required by your environment as described in the “Overview of the Fix Deployment Process” section of this document.
python /usr/ibmpacks/bin/3.0-SNAPSHOT/upgrade/bigsql_upgrade.py -m Restore

The Restore phase will take about 10 minutes. This time may vary depending on the specs of the machines & the number of Big SQL objects (tables, etc ...).

At the end of the Restore phase, the new Big SQL service will be back to its initial state (before theBackup phase was run). It will show in the Ambari dashboard and the fix is not installed.

In case of failure reported by the script during the Restore phase, identify the cause of the issue (access control, lack of disk space, etc ...), resolve it and re-run the Restore phase.
The Restore phase is designed to be re-run as many times as needed.