SevOne Data Insight Upgrade Process Guide

1. Using ssh , log into SevOne Data Insight as sevone.

   
   $ ssh sevone@<SevOne Data Insight 'control plane' node IP address or hostname>
   

   
   

   Example: Currently on SevOne Data Insight 6.8 and to be upgraded to SevOne Data Insight 7.0

   
   $ ssh sevone@10.49.13.240

2. Change directory to /opt/SevOne/upgrade.

   
   $ cd /opt/SevOne/upgrade
   

3. Download the following (latest) files from IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact IBM SevOne Support for the files. You must place these files in /opt/SevOne/upgrade directory of the control plane node.
   1. sdi-v7.0.0-build.<latest>.tgz
   2. sdi-v7.0.0-build.<latest>.tgz.sha256.txt
   3. signature-tools-<latest version>-build.<###>.tgz For example, signature-tools-2.0.3-build.1.tgz
   4. signature-tools-<latest version>-build.<###>.tgz.sha256.txtFor example, signature-tools-2.0.3-build.1.tgz.sha256.txt

      
      $ ls -lah /opt/SevOne/upgrade
      

      You should see all the files mentioned above in this directory.

      Note: /opt/SevOne/upgrade/freight directory contains yum packages for air-gapped (no internet) environments.

Once you have all the required files, execute the commands below.

1. Verify the checksum of the code signing tool before extracting it.

   
   $ cd /opt/SevOne/upgrade
   
   $ cat $(ls -Art /opt/SevOne/upgrade/signature-tools-*.tgz.sha256.txt | \
   tail -n 1) | sha256sum --check
   
   $ sudo tar xvfz $(ls -Art /opt/SevOne/upgrade/signature-tools-*.tgz | \
   tail -n 1) -C /
   
   $ cd ~
   

2. Verify the signature of SevOne Data Insight .tgz files.

   
   $ /usr/local/sbin/SevOne-validate-image \
   -i $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz | tail -n 1) \
   -s $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz.sha256.txt | tail -n 1)
   

3. Remove old upgrade files, if any.

   
   $ find /opt/SevOne/upgrade -maxdepth 1 \
   -type d -not -name "upgrade" | xargs rm -rf
   

4. Extract the latest SevOne Data Insight build files.

   
   $ tar xvf $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz  | \
   tail -n 1) -C /opt/SevOne/upgrade/
   

5. You may now install / upgrade SOA.
   Attention:
   This step only applies if your SevOne NMS is on 6.x and you want to upgrade SOA.
   
   If your SevOne NMS is on 7.0 or above, please skip this step.
   Note:

   • When upgrading SOA, nginx service will restart on your SevOne NMS. This causes a very brief service impact.
   • SOA version
     SOA must be on the latest version on all appliances in SevOne NMS cluster. Command Line Interface (CLI) must be used to upgrade SOA on all peers as the graphical user interface (GUI) only upgrades SOA for the NMS appliance you are connected to.
   
   

   Execute the command to install / upgrade SOA only on the Cluster Master and HSA

   Warning: Cluster Master must be a PAS. It cannot be a DNC.
   
   Ensure that SevOne Data Insight does not point to a peer that is a DNC.

   
   $ sevone-cli soa upgrade \
   /opt/SevOne/upgrade/utilities/SevOne-soa-*.rpm
   

   Execute the command to install / upgrade SOA on ALL peers in the cluster

   
   $ sevone-cli soa upgrade \
   /opt/SevOne/upgrade/utilities/SevOne-soa-*.rpm \
   --all-peers
   

6. Run the pre-check playbook.

   
   $ sevone-cli playbook precheck
   

7. Ensure ports 3000 and 3001 are accessible from your browser. The following table lists the default port numbers and its respective config file location for the client and API.

   Name	Default Port	Config File Location
   Client	3000	/etc/sevone-guii/client.custom.yaml
   API	3001	/etc/sevone-guii/api.custom.yaml

If you are upgrading from SevOne Data Insight 6.7.x or below, your operating system will be migrated from CentOS 7 to RHEL 8. This happens as part of the upgrade automatically. Your machine will restart multiple times. If you are upgrading using Graphical User Interface, you can track the progress as each playbook is run. If you are upgrading using Command Line Interface, you can track progress in journalctl.

SevOne Data Insight now programmatically enforces the production specifications of of 8 vCPU cores / 32 GB RAM / 250 GB HDD.

Please refer to SevOne Data Insight Administration Guide > section Configuration > subsection Terrraform Module > Resource Requests & Limits for details on configuring the resource limits or disabling this requirement.

Kubernetes creates self-signed certificates to communicate with its API Server. It is possible for these certificates to expire before they are auto-rotated.

If you see the following error when running any kubectl commands, please renew your Kubernetes certificates before continuing with the installation process. For details, please refer to SevOne Data Insight Administration Guide > section Deployment > subsection Rotate Kubernetes Certificates.

$ kubectl get pods
Unable to connect to the server: x509: certificate has expired or is not yet valid

$ sudo yum remove -y expect tcl

You must be in screen session to execute the upgrade command below.

Start 'screen' session

$ screen -S upgrade

When running the upgrade using the Command Line Interface, it validates the checksum, extracts the upgrade file, upgrades sevone-cli (if required), upgrades SOA, attempts to read NMS IP list from SevOne Data Insight database, and configures the SSH keys for the NMS being used as a datasource.

Update SevOne Data Insight without using the Graphical User Interface Installer

$ /usr/local/sbin/SevOne-validate-image \
-i $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz | tail -n 1) \
-s $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz.sha256.txt | tail -n 1) \
--installer sdi

Wait for a few minutes before you launch SevOne Data Insight.

Using a web browser of your choice, enter your SevOne Data Insight IP address or hostname of the machine you have upgraded in the URL. You will notice that SevOne Data Insight is now on the upgraded version i.e., 7.0.0.

1. Execute the following command to update SOA on all your datasources and configure the Graphical User Interface Installer. You must be in screen session to execute the upgrade command below.

   Start 'screen' session

   
   $ screen -S upgrade
   

   After the upgrade completes successfully, SevOne highly recommends a reboot to update the kernel as it does not update automatically.
   
   To reboot, execute the following command.

   
   $ sudo reboot
   

   Please wait for ~10-15 minutes for Kubernetes and other services to fully spin up.

   Update SevOne Data Insight using the Graphical User Interface Installer

   
   $ /usr/local/sbin/SevOne-validate-image \
   -i $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz | tail -n 1) \
   -s $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz.sha256.txt | tail -n 1) \
   --installer sdi-gui
   

   Note: When running the command above, you may notice some text in RED in some of the playbooks. If it says ...ignoring after this text, it is a benign error and it can be ignored.

   Example: The command returns the following

   
   ╒══════════════════════════════════════════════════════════════╕
   │ SEVONE GUI INSTALLER                                         │
   ╞══════════════════════════════════════════════════════════════╡
   │ Please open https://10.49.13.240:3000 in your web browser to │
   │ access the GUI Installer.                                    │
   ├──────────────────────────────────────────────────────────────┤
   │ Your credentials are:                                        │
   │ - Username: admin                                            │
   │ - Password: LPQoYBQ|p|                                       │
   ├──────────────────────────────────────────────────────────────┤
   │ If you ever lose your credentials, they're stored in:        │
   │ /etc/sevone-guii/creds                                       │
   ╘══════════════════════════════════════════════════════════════╛

   You are now ready to upgrade SevOne Data Insight using the Graphical User Interface Installer.

   Important: Remove packages
   You may have expect and tcl in your environment. Prior to starting the upgrade, please execute the following command to remove these packages.

   
   $ sudo yum remove -y expect tcl
   

2. Using a web browser of your choice, enter the URL returned above. For example, https://10.49.13.240:3000.
   
   Note: You will also need the credentials (Username and Password) returned. These credentials are also stored in /etc/sevone-guii/creds file.

   Example

   
   $ cat /etc/sevone-guii/creds | jq
   {
     "password": "LPQoYBQ|p|",
     "tokenSecret": "AWfBRZQRnitEELRgYpsZXJnslHcdvUnV",
     "username": "admin"
   }

3. Click Update Cluster to update SevOne Data Insight. For example, upgrade from SevOne Data Insight 6.8.0+46 to SevOne Data Insight 7.0.0+316.
4. Enter the credentials returned to perform the Self-Service Upgrade. For example, Username: admin and Password: LPQoYBQ|p|
   
   Note:
   To use the Graphical User Interface installer in dark theme, click next to SevOne logo.
   For help on what each upgrade step does, click button in the upper-right corner.
   Important: All the screenshots below are based on the example being used to write this document. Your total number of tasks passed (ok), skipped, failed, ignored, unreachable, or unexecuted will vary based on your setup. The tasks failed must be addressed as ansible has not ignored them.
5. The graphical user interface installer checks the Current Version and the Upgrade Available version. If the current version is older than the upgrade version available, you are now ready to continue to Pre-Upgrade.
   
   Important: Example
   Current Version is on SevOne Data Insight 6.8.0+46.
   Upgrade Available version is SevOne Data Insight 7.0.0+316.
   
   Important: During the Self-Service Upgrade, if you experience network connectivity issue or the upgrade has been halted for any reason, the self-service upgrade will resume from the step where it left off after the issue is resolved. However, if you are at the Upgrade step and the self-service upgrade has been halted for any reason, self-service upgrade will show a message requesting you to contact SevOne Support.
   To resume with the Self-Service Upgrade, using a web browser of your choice, re-enter the URL the setup script has returned. For example, https://10.49.13.240:3000.
6. Click Continue to Pre-Upgrade>button.
7. You are now ready to run the pre-upgrade. Click Run Pre-Upgrade button.
   Pre-Upgrade step runs various checks to ensure that SevOne Data Insight cluster is healthy before the upgrade. It takes backups of the database and creates a copy of the security keys.
   To view the logs for a task, click for the task you need the details for. The pop-up has Copy to clipboard button which allows you to copy all the contents in the pop-up and paste it into a file.
8. Click Continue button to upgrade SOA.
   
   Note: SOA version
   SOA must be on the latest version on all appliances in SevOne NMS cluster.
   
   If your peers are on SevOne NMS 7.x or above, skip the following command. However, if your peers are on SevOne NMS 6.x,

   • Command Line Interface (CLI) must be used to upgrade SOA on all peers as the graphical user interface (GUI) only upgrades SOA for the NMS appliance you are connected to.
   • Add flag --all-peers if you want to install SOA on all peers in the cluster.

   
   $ sevone-cli soa upgrade \
   /opt/SevOne/upgrade/utilities/SevOne-soa-*.rpm \
   <enter SevOne NMS IP address> --all-peers
   

9. You are now ready to upgrade SOA. Click Run Upgrade SOA button. This can take a few minutes to run.
   
10. Important: For SevOne Data Insight 7.0, click Continue to go to step 4, Configure and click Continue again to skip the configuration as it does not apply for this release.
11. Click Continue button to upgrade.
12. You are now at the Upgrade step.
    
    
13. Click Run Upgrade button to run the upgrade.
    
    Important: Allows you to start the upgrade and show the progress.
    If you are upgrading from SevOne Data Insight 6.7 or lower, it will upgrade your operating system from CentOS 7 to RHEL 8 and can take over an hour to run.
    
    
    SevOne Data Insight upgrade has now completed.
    Note: The task list can be long. Search tasks capability is available to search for text you are looking for in the task list.
    Note: To view the logs for a task, click for the task you need the details for. The pop-up has Copy to clipboard button which allows you to copy all the contents in the pop-up and paste it into a file.
14. Click the Continue button.
    
    Important: This indicates that the upgrade has completed successfully.
15. You may now link the link Click Here.
16. Link Click Here will launch SevOne Data Insight. You will notice that SevOne Data Insight is now on the upgraded version i.e., 7.0.0.
    

Example: Upgrade not required
If you are already on the version you are trying to upgrade to, you will get the You can proceed with redeploy/install message. This allows you to upgrade to a later version, if available.

Important: Additional details on GUI Installer can be found in SevOne Data Insight Administration Guide > Tools > section GUI Installer.

Note: Please refer to SevOne Data Insight Administration Guide for configuration and deployment details and SevOne Data Insight Troubleshooting / Debug / Tools Guide for troubleshooting scenarios, how to debug SevOne Data Insight, and various tools available.

In addition to the regular pre-upgrade checklist, additional checks must be considered when migrating the operating systems.

The regular upgrade process will automatically detect if an operating system migration from CentOS to RHEL 8 is required.

Several restarts will occur and it may take an hour for a single node or 2 hours for multi-node.

$ journalctl -u reboot2continue -f

July 03 21:25:08 sdi-node02 systemd[1]: reboot2continue.service: Succeeded.
July 03 21:25:08 sdi-node02 systemd[1]: Started Upgrade SevOne on boot.

TASK [leapp2rhel8 : Install Python requests library]

This task can fail after a stalled upgrade and the yum cache growing stale.

Execute the following commands to resolve the issue.

$ ansible all --become -a 'warn=false sudo yum clean all'
$ ansible all --become -a 'warn=false yum install -y python-requests --disablerepo=* --enablerepo=freight'

TASK [leapp2rhel8 : Confirm pam_pkcs11 module check]

Mostly seen on older appliances that have been upgraded from SevOne Data Insight 3.4 or below.

This error is caused by the Python environment not detecting a dependent package which can be reinstalled by running the following commands.

$ ansible all --become -a 'warn=false rm -rf /var/freight/*'
$ ansible all --become -m unarchive -a 'src=/opt/SevOne/upgrade/freight/packages/rhel7-update-09-02-2024-1707496657.el7.tgz dest=/var/freight'
$ ansible all --become -a 'warn=false sudo yum clean all'
$ ansible all --become -a 'warn=false yum reinstall -y python-requests --disablerepo=* --enablerepo=freight'

TASK [leapp upgrade]

This error is usually caused by the host system that still contains the extraneous packages.

$ sudo leapp upgrade --no-rhsm --no-insights-register --iso /opt/SevOne/upgrade/freight/packages/leapp2rhel8-*.iso

============================================================
                           ERRORS
============================================================

2024-01-26 14:39:52.712457 [ERROR] Actor: dnf_package_download
Message: DNF execution failed with non zero exit code.
Summary:
    Stderr: Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
            Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
            No matches found for the following disable plugin patterns: subscription-manager
            Warning: Package marked by Leapp to upgrade not found in repositories metadata: gpg-pubkey leapp-upgrade-el7toel8 leapp python2-leapp
            Error opening /iso/BaseOS/Packages/t/tcl-8.6.8-2.el8.x86_64.rpm: No such file or directory
            Package "tcl-1:8.6.8-2.el8.x86_64" from local repository "BaseOS" has incorrect checksum
            Error opening /iso/BaseOS/Packages/e/expect-5.45.4-5.el8.x86_64.rpm: No such file or directory
            Package "expect-5.45.4-5.el8.x86_64" from local repository "BaseOS" has incorrect checksum
            Error: Some packages from local repository have incorrect checksum