Question & Answer
Question
How to deploy a patch for BigInsights Big SQL 4.2.5?
Answer
Introduction
This document describes the patch management process for BigInsights Big SQL 4.2.5
Table of Contents
Introduction
Requirements
Overview of the Fix Deployment Process
Example invocation of the script
Script Logging & Troubleshooting
Fix Installation Instructions
Obtaining the Fix
Setting up a Local Repo Site
Install the BigInsights Value-Add RPM
Running the Backup phase of the Fix Deployment Process
Running the Upgrade phase of the Fix Deployment Process
Running the Finalize phase of the Fix Deployment Process
Running the Restore phase of the Fix Deployment Process
Requirements
The following information will be required to run the patch management process:
• The protocol (http/https), host name & port number of the Apache Ambari server for the cluster
• The Apache Ambari administrator user name and password
• If the cluster has Kerberos security enabled, the Kerberos administrator credentials (principal & password)
In addition, to upgrade Big SQL, there must exist a user with the following attributes:
• passwordless sudo access on all nodes of the cluster, including the Ambari server itself.
• The ability to connect passwordlessly through ssh from the Ambari server to all Big SQL nodes.
This user can be root. If the user is not root, the user name must be passed to the upgrade script with the -a option. The upgrade script must be run with root user privilege. This can be achieved by using the sudo command.
The current patch management process requires about 4 Gb of free disk space on the root (/) partition. This space is temporarily used by the installation process and released once the installation is complete. In addition to this, the patch management process also requires about 2GB of free disk space on /usr for DB2 installation. If above mentioned spaces are not available, then the patch management prerequisite checker will warn you and stop the installation process.
Apache Ambari configuration groups are not supported by the patch management process. If configuration groups are present, the patch management prerequisite checker will warn you and stop the installation process. It is recommended to disable / remove configuration groups before initiating the patch management process.
The HDFS service must be up and running in the Apache Ambari server. If it is not running, the patch management prerequisite checker will warn you and stop the installation process.
The Big SQL High Availability (HA) feature must be disabled on the cluster, and there must be only one head node for the installation to proceed. If HA is enabled, the patch management prerequisite checker will warn you and stop the installation process.
Overview of the Fix Deployment Process
The patch installation process is a manual, scripted process which consists of three distinct phases that must be run in sequence by executing the bigsql_upgrade.py script with the appropriate options. The three phases are Backup, Upgrade and Finalize.
During the upgrade process, the state of the BigInsights -Big SQL service might be reported incorrectly in the Apache Ambari server web interface. This is expected behavior. It will return to normal after the upgrade is complete.
The script to install the patch is located at /usr/ibmpacks/scripts/1.2.5.0/upgrade/BIGSQL/bigsql_upgrade.py on the Ambari server host after the BigInsights value-add package is deployed on the system.
The script accepts the following options:
-m phase
Required -The upgrade phase to run. Valid values are: Backup, Upgrade, Finalize, or Restore.
-u ambari admin user
Optional -The Ambari admin user name. This defaults to admin.
-p ambari admin pwd
Optional -Ambari admin password. This defaults to admin.
-s http protocol
Optional -Ambari server protocol ('http' or 'https') . This defaults to http.
-P ambari server port
Optional -Ambari server port number. This defaults to 8080.
-k kerberos admin principal
Optional -Kerberos admin principal. This is required if Kerberos is enabled on the cluster.
-w kerberos admin password
Optional -Kerberos admin password. This is required if Kerberos is enabled on the cluster.
-a ssh user
Optional -Identity used for inter-node passwordless ssh connections. This defaults to root.
Example invocation of the script
For this example, assume the following criteria:
• The Ambari dashboard is https://ambari.company.com:9443/.
• The administrator user name is 'ambadmin' and the password is 'ambpassword'.
• Kerberos is not enabled.
• The environment is set up for passwordless ssh connectivity between nodes for a user named 'ssh_user'. This user also has passwordless sudo rights.
• You are logged in as the root user.
To run the 'Backup' phase of the Big SQL service patch management process on your BigInsights cluster, invoke the script in the following way
/usr/ibmpacks/scripts/1.2.5.0/upgrade/BIGSQL/bigsql_upgrade.py -u ambadmin -p ambpassword -s https -P 9443 -a ssh_user -m Backup
Script Logging & Troubleshooting
A complete log of the script execution is located at /var/ibm/bigsql/logs/upgrade.log on the Ambari server node (the one where the script is executed).
Logs for additional sub tasks performed by the script (such as package installation) on individual nodes of the cluster are located in the /var/ibm/bigsql/logs directory on the individual cluster (head and worker) nodes.
Additional logs related to the upgrade of the Big SQL engine will be found in the /tmp and /tmp/bigsql/logs directory on the Big SQL head node.
Fix Installation Instructions
Obtaining the Fix
When a defect is reported and fixed, a new BigInsights Value-Add build containing the fix will be generated and provided to you as a TAR archive by IBM Technical Support. This archive must be used to set up a repo site in a network location accessible by all the nodes in the BigInsights cluster. This process is very similar to performing an OFFLINE installation of BigInsights and is detailed below.
Setting up a Local Repo Site
The following steps will walk you through the process of creating a local repository to make the fix installation files available to the nodes in your cluster. This must be done on a network node that can be accessed by all nodes in the BigInsights cluster. If you do not have an additional Linux host, you can use one of the Hadoop management nodes. The repository server requires an HTTP web server. The steps below explain how to install it.
1. On the repository server, if the Apache HTTP server is not installed, install it:
sudo yum install httpd
2. On the repository server, ensure that the createrepo package is installed.
sudo yum install createrepo
3. Make sure there is network access from all nodes in your cluster to the repository server. If data nodes are on a private network and the repository server is external to the Hadoop cluster, you might need to configure iptables for IP forwarding and masquerading.
4. On the repository server, create a directory to host your value-add repository, such as <mirror web server document root>/repos/valueadds. For example, for Apache httpd, the default document root is /var/www/html, so the command to run is
sudo mkdir -p /var/www/html/repos/valueadds
5. Expand the TAR archive received from IBM Technical Support into the mirror web server location, <your.mirror.web.server.document root>/repos/valueadds directory. Note that their exact names will depend on the platform & OS family you cluster is running on.
sudotar -xvzf biginsights_iop_4.2.5.0.el6.x86_64.tar.gz -C /var/www/html/repos/valueadds/
6. Start the web server. If you use Apache httpd, start it by using either of the following commands:
sudo apachectl start
or
sudo service httpd start
7. Ensure that any firewall settings allow inbound HTTP access from your cluster nodes to the mirror web server.
8. Test your local repository by browsing to the web directory. Note that the exact path depends on the OS & platform your cluster is running on. Please substitute as appropriate.
http://<your.mirror.web.server>/repos/valueadds/BigInsights-Valuepacks/RHEL6/x86_64/4.2.5.0/
9. You should see all of the files that you copied to the repository server. On the repository server, run the createrepo command to initialize the repository, again substituting as appropriate.
createrepo /var/www/html/repos/valueadds/BigInsights-Valuepacks/RHEL6/x86_64/4.2.5.0/
Install the BigInsights Value-Add RPM
1. Locate the BigInsights Value-Add RPM to install on the Ambari Server host of the Hadoop cluster in the /var/www/html/repos/valueadds/BigInsights-Valuepacks/RHEL6/x86_64/4.2.5.0/ directory –note, that the name may be different depending on the OS family & platform your cluster is running on
BigInsights-IOP-1.2.0.0-4.2.<family>.<platform>.rpm
2. Copy this RPM file to the Ambari Server host and install it by using the following commands:
sudo yum localinstall BigInsights-IOP-1.2.0.0-4.2.<family>.<platform>.rpm
3. On the Ambari Server node, navigate to the /var/lib/ambari-server/resources/stacks/BigInsights/4.2.5/repos/repoinfo.xml file. Ensure the <baseurl> element for the BIGINSIGHTS-VALUEPACK-1.2.5.0<repo> entry points to your repository server. Remember, there might be multiple<repo> sections. Make sure that the URL you tested in step 8 matches exactly the value indicated in the <baseurl> element. For example, the repoinfo.xml might look like the following after your changes are applied (the part in bold indicating the changes that you have performed):
...
<repo>
<baseurl>http://<your.mirror.web.server>/repos/valueadds/BigInsights-Valuepacks/RHEL6/x86_64/4.2.5.0</baseurl>
<repoid>BIGINSIGHTS-VALUEPACK-1.2.5.0</repoid>
<reponame>BIGINSIGHTS-VALUEPACK-1.2.5.0</reponame>
</repo>
...
4. Note: The new <repo> section might appear as a single line.
5. Tip: If you later find an error in this configuration file, make corrections and run the following command:
yum clean all
6. Restart the Ambariserver.
ambari-server restart
7. Give the Ambari server a few minutes to fully replicate its updated configuration to all its nodes after the restart. On small clusters (less than 5 nodes) a minute after the server has restarted should be enough. On bigger cluster, it is recommended to wait about 10 minutes.
Running the Backup phase of the Fix Deployment Process
The “Backup” phase is responsible for backing up on all Big SQL nodes the Big SQL catalogue, configuration information and data for the pure DB2 tables (i.e. tables that are not Hadoop or HBase tables). Please note that the data for Hadoop & HBase tables reside on HDFS and is not touched by the fix deployment process.
To perform the Backup phase, execute the Big SQL upgrade script with the -m Backup option. Add any other options required by your environment as described in the “Overview of the Fix Deployment Process” section of this document.
python /usr/ibmpacks/scripts/1.2.5.0/upgrade/BIGSQL/bigsql_upgrade.py -m Backup
The Backup phase will take about 15 minutes. This time may vary depending on the specs of the machines & the bandwidth of the link between the repo site and the cluster nodes. On successful completion of the Backup phase, the script will issue the following message:
At the end of the Backup phase, the Big SQL catalogue, configuration information and data for the pure DB2 tables are backed up. The Big SQL service is NOT RUNNING and DOES NOT show in the Ambari dashboard. This is the expected behavior. If needed, Big SQL service can be started from the command line and used. Please note that in that case, the fix is not effective yet.
In case of failure reported by the script during the Backup phase, identify the cause of the issue (access control, lack of disk space, etc...), resolve it and re-run the backup phase. The backup phase is designed to be run repeatedly as many times as necessary.
Running the Upgrade phase of the Fix Deployment Process
The “Upgrade” phase is responsible for installing the Big SQL binaries containing the fix, and perform any additional management task required by the fix.
To perform the Upgrade phase, execute the Big SQL upgrade script with the -m Upgrade option. Add any other options required by your environment as described in the “Overview of the Fix Deployment Process” section of this document.
python /usr/ibmpacks/scripts/1.2.5.0/upgrade/BIGSQL/bigsql_upgrade.py-m Upgrade
The Upgrade phase will take about 25 minutes. This time may vary depending on the specs of the machines & the number of Big SQL objects (tables, etc...).On successful completion of the Upgrade phase, the script will issue the following message:
At the end of the Upgrade phase, the new Big SQL binaries are deployed. The Big SQL service is RUNNING and DOES NOT show in the Ambari dashboard. This is the expected behavior. The Big SQL service can be used to run some test, evaluate and validates from the command line or through a standard JDBC connection as the fix is now effective.
In case of failure reported by the script during the Upgrade phase, DO NOT re-run the Upgrade phase (the script will prevent it). Instead, identify the cause of the issue (access control, lack of disk space, etc...), resolve it and run the Restore phase as explained in the “Running the Restore phase of the Fix Deployment Process” section of this document.
Running the Finalize phase of the Fix Deployment Process
IMPORTANT NOTE: Once the Finalize phase has run, the fix installation is definitive and cannot be reverted. Back-ups created during the backup phase are also cleaned up. It is recommended to verify that the installation of the fix has been successful by running test, evaluate and validates before running the Finalize phase.
The “Finalize” phase is responsible for finalizing the installation and cleaning up backups and other work files created during the fix deployment process.
To perform the Finalize phase, execute the Big SQL upgrade script with the -m Finalize option. Add any other options required by your environment as described in the “Overview of the Fix Deployment Process” section of this document.
python /usr/ibmpacks/scripts/1.2.5.0/upgrade/BIGSQL/bigsql_upgrade.py-m Finalize
The Finalize phase will take about 3 minutes. This time may vary depending on the specs of the machines. On successful completion of the Finalize phase, the script will issue the following message:
At the end of the Finalize phase, the Big SQL service is running and showing in the Ambari dashboard. The fix is now deployed and the Big SQL service is ready to use again.
In case of failure reported by the script during the Finalize phase, identify the cause of the issue (access control, lack of disk space, etc...), resolve it and re-run the Finalize phase. The Finalize phase is designed to be run repeatedly as many times as necessary.
Running the Restore phase of the Fix Deployment Process
IMPORTANT NOTE: This phase must only be run in case of failure (in the Upgrade phase) or to remove a defective fix.
In case of failures of the Upgrade phase, it is necessary to restore the backup to bring back the cluster to a state where the Upgrade phase can be re-attempted. The Restore phase can also be used to remove a fix that is proving defective, assuming that the Finalize phase has not been run yet.
The “Restore” phase is responsible for restoring the backups taken during the Backup phase and bringing back the cluster to its initial state before the Backup phase was run.
To perform the Restore phase, execute the Big SQL upgrade script with the -m Restore option. Add any other options required by your environment as described in the “Overview of the Fix Deployment Process” section of this document.
python /usr/ibmpacks/scripts/1.2.5.0/upgrade/BIGSQL/bigsql_upgrade.py-m Restore
The Restore phase will take about 10 minutes. This time may vary depending on the specs of the machines & the number of Big SQL objects(tables, etc...).On successful completion of the Restore phase, the script will issue the following message:
At the end of the Restore phase, the Big SQL service will be back to its initial state (before the Backup phase was run). It will show in the Ambari dashboard and the fix is not installed.
In case of failure reported by the script during the Restore phase, identify the cause of the issue (access control, lack of disk space, etc...), resolve it and re-run the Restore phase. The Restore phase is designed to be run repeatedly as many times as necessary.
Related Information
[{"Product":{"code":"SSCRJT","label":"IBM Db2 Big SQL"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Big SQL","Platform":[{"code":"PF016","label":"Linux"}],"Version":"4.2.5","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]
Was this topic helpful?
Document Information
Modified date:
18 July 2020
UID
swg22008143