SevOne Data Insight Upgrade Process Guide

In addition to the regular pre-upgrade checklist, additional checks must be considered when migrating the operating systems.

The regular upgrade process will automatically detect if an operating system migration from CentOS to RHEL 8 is required.

Several restarts will occur and it may take an hour for a single node or 2 hours for multi-node.

$ journalctl -u reboot2continue -f

May 03 21:25:08 sdi-node02 systemd[1]: reboot2continue.service: Succeeded.
May 03 21:25:08 sdi-node02 systemd[1]: Started Upgrade SevOne on boot.

TASK [leapp2rhel8 : Install Python requests library]

This task can fail after a stalled upgrade and the yum cache growing stale.

Execute the following commands to resolve the issue.

$ ansible all --become -a 'warn=false sudo yum clean all'
$ ansible all --become -a 'warn=false yum install -y python-requests --disablerepo=* --enablerepo=freight'

TASK [leapp2rhel8 : Confirm pam_pkcs11 module check]

Mostly seen on older appliances that have been upgraded from SevOne Data Insight 3.4 or below.

This error is caused by the Python environment not detecting a dependent package which can be reinstalled by running the following commands.

$ ansible all --become -a 'warn=false rm -rf /var/freight/*'
$ ansible all --become -m unarchive -a 'src=/opt/SevOne/upgrade/freight/packages/rhel7-update-09-02-2024-1707496657.el7.tgz dest=/var/freight'
$ ansible all --become -a 'warn=false sudo yum clean all'
$ ansible all --become -a 'warn=false yum reinstall -y python-requests --disablerepo=* --enablerepo=freight'

TASK [leapp upgrade]

This error is usually caused by the host system that still contains the extraneous packages.

$ sudo leapp upgrade --no-rhsm --no-insights-register --iso /opt/SevOne/upgrade/freight/packages/leapp2rhel8-*.iso

============================================================
                           ERRORS
============================================================

2024-01-26 14:39:52.712457 [ERROR] Actor: dnf_package_download
Message: DNF execution failed with non zero exit code.
Summary:
    Stderr: Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
            Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
            No matches found for the following disable plugin patterns: subscription-manager
            Warning: Package marked by Leapp to upgrade not found in repositories metadata: gpg-pubkey leapp-upgrade-el7toel8 leapp python2-leapp
            Error opening /iso/BaseOS/Packages/t/tcl-8.6.8-2.el8.x86_64.rpm: No such file or directory
            Package "tcl-1:8.6.8-2.el8.x86_64" from local repository "BaseOS" has incorrect checksum
            Error opening /iso/BaseOS/Packages/e/expect-5.45.4-5.el8.x86_64.rpm: No such file or directory
            Package "expect-5.45.4-5.el8.x86_64" from local repository "BaseOS" has incorrect checksum
            Error: Some packages from local repository have incorrect checksum

1. Using ssh , log into SevOne Data Insight as sevone.

   
   $ ssh sevone@<SevOne Data Insight 'control plane' node IP address or hostname>
   

   
   

   Example: Currently on SevOne Data Insight 6.7 and to be upgraded to SevOne Data Insight 6.8

   
   $ ssh sevone@10.49.14.243
   

2. Change directory to /opt/SevOne/upgrade.

   
   $ cd /opt/SevOne/upgrade
   

3. Download the following (latest) files from IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact IBM SevOne Support for the files. You must place these files in /opt/SevOne/upgrade directory of the control plane node.
   1. sdi-v6.8.0-build.<latest version>.tgz
   2. sdi-v6.8.0-build.<latest version>.tgz.sha256.txt
   3. signature-tools-<latest version>-build.<###>.tgz For example, signature-tools-2.0.2-build.1.tgz
   4. signature-tools-<latest version>-build.<###>.tgz.sha256.txt For example, signature-tools-2.0.2-build.1.tgz.sha256.txt

      
      $ ls -lah /opt/SevOne/upgrade
      

      You should see all the files mentioned above in this directory.

      Note: /opt/SevOne/upgrade/freight directory contains yum packages for air-gapped (no internet) environments.

Once you have all the required files, execute the commands below.

1. Verify the checksum of the code signing tool before extracting it.

   
   $ cd /opt/SevOne/upgrade
   
   $ cat $(ls -Art /opt/SevOne/upgrade/signature-tools-*.tgz.sha256.txt | \
   tail -n 1) | sha256sum --check
   
   $ sudo tar xvfz $(ls -Art /opt/SevOne/upgrade/signature-tools-*.tgz | \
   tail -n 1) -C /
   
   $ cd ~
   

2. Verify the signature of SevOne Data Insight .tgz files.

   
   $ /usr/local/sbin/SevOne-validate-image \
   -i $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz | tail -n 1) \
   -s $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz.sha256.txt | tail -n 1)
   

3. Remove old upgrade files, if any.

   
   $ find /opt/SevOne/upgrade -maxdepth 1 \
   -type d -not -name "upgrade" | xargs rm -rf
   

4. Extract the latest SevOne Data Insight build files.

   
   $ tar xvf $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz  | \
   tail -n 1) -C /opt/SevOne/upgrade/
   

5. You may now install / upgrade SOA.
   Important:
   When upgrading SOA, nginx service will restart on your SevOne NMS. This causes a very brief service impact.
   Note: SOA version
   SOA must be on the latest version on all appliances in SevOne NMS cluster. Command Line Interface (CLI) must be used to upgrade SOA on all peers as the graphical user interface (GUI) only upgrades SOA for the NMS appliance you are connected to.
   
   

   Execute the command to install / upgrade SOA only on the Cluster Master and HSA

   
   $ sevone-cli soa upgrade \
   /opt/SevOne/upgrade/utilities/SevOne-soa-*.rpm
   

   Execute the command to install / upgrade SOA on ALL peers in the cluster

   
   $ sevone-cli soa upgrade \
   /opt/SevOne/upgrade/utilities/SevOne-soa-*.rpm \
   --all-peers
   

6. Run the pre-check playbook.

   
   $ sevone-cli playbook precheck
   

7. Ensure ports 3000 and 3001 are accessible from your browser. The following table lists the default port numbers and its respective config file location for the client and API.

   Name	Default Port	Config File Location
   Client	3000	/etc/sevone-guii/client.custom.yaml
   API	3001	/etc/sevone-guii/api.custom.yaml

If you are upgrading from SevOne Data Insight 6.7.x or below, your operating system will be migrated from CentOS 7 to RHEL 8. This happens as part of the upgrade automatically. Your machine will restart multiple times. If you are upgrading using Graphical User Interface, you can track the progress as each playbook is run. If you are upgrading using Command Line Interface, you can track progress in journalctl.

SevOne Data Insight now programmatically enforces the production specifications of of 8 vCPU cores / 32 GB RAM / 250 GB HDD.

Please refer to SevOne Data Insight Administration Guide > section Configuration > subsection Helm Chart > Resource Requests & Limits for details on configuring the resource limits or disabling this requirement.

Kubernetes creates self-signed certificates to communicate with its API Server. It is possible for these certificates to expire before they are auto-rotated.

If you see the following error when running any kubectl commands, please renew your Kubernetes certificates before continuing with the installation process. For details, please refer to SevOne Data Insight Administration Guide > section Deployment > subsection Rotate Kubernetes Certificates.

$ kubectl get pods
Unable to connect to the server: x509: certificate has expired or is not yet valid

The upgrade duration increases alongside the number of pod replicas (particularly with GraphQL pods). By default, the upgrade times out after 5 minutes; after which the rollback begins. You must consider increasing the timeout by specifying a duration for the timeout setting under app_di.

Using a text editor of your choice, edit /etc/ansible/group_vars/all/di.yaml file to change the timeout variable under app_di and then, save it.

Configure Timeout

app_di:
  chart: di
  enabled: true
  name: di
  namespace: default
  timeout: 10m

$ sudo yum remove -y expect tcl

You must be in screen session to execute the upgrade command below.

Start 'screen' session

$ screen -S upgrade

When running the upgrade using the Command Line Interface, it validates the checksum, extracts the upgrade file, upgrades sevone-cli (if required), upgrades SOA, attempts to read NMS IP list from SevOne Data Insight database, and configures the SSH keys for the NMS being used as a datasource.

Update SevOne Data Insight without using the Graphical User Interface Installer

$ /usr/local/sbin/SevOne-validate-image \
-i $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz | tail -n 1) \
-s $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz.sha256.txt | tail -n 1) \
--installer sdi

Wait for a few minutes before you launch SevOne Data Insight.

Using a web browser of your choice, enter your SevOne Data Insight IP address or hostname of the machine you have upgraded in the URL. You will notice that SevOne Data Insight is now on the upgraded version i.e., 6.8.0.

1. Execute the following command to update SOA on all your datasources and configure the Graphical User Interface Installer. You must be in screen session to execute the upgrade command below.

   Start 'screen' session

   
   $ screen -S upgrade
   

   After the upgrade completes successfully, SevOne highly recommends a reboot to update the kernel as it does not update automatically.
   
   To reboot, execute the following command.

   
   $ sudo reboot
   

   Please wait for ~10-15 minutes for Kubernetes and other services to fully spin up.

   Update SevOne Data Insight using the Graphical User Interface Installer

   
   $ /usr/local/sbin/SevOne-validate-image \
   -i $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz | tail -n 1) \
   -s $(ls -Art /opt/SevOne/upgrade/sdi-*.tgz.sha256.txt | tail -n 1) \
   --installer sdi-gui
   

   Note: When running the command above, you may notice some text in RED in some of the playbooks. If it says ...ignoring after this text, it is a benign error and it can be ignored.

   Example: The command returns the following

   
   ╒══════════════════════════════════════════════════════════════╕
   │ SEVONE GUI INSTALLER                                         │
   ╞══════════════════════════════════════════════════════════════╡
   │ Please open https://10.49.14.243:3000 in your web browser to │
   │ access the GUI Installer.                                    │
   ├──────────────────────────────────────────────────────────────┤
   │ Your credentials are:                                        │
   │ - Username: admin                                            │
   │ - Password: }_G<{oJ|`!                                    │
   ├──────────────────────────────────────────────────────────────┤
   │ If you ever lose your credentials, they're stored in:        │
   │ /etc/sevone-guii/creds                                       │
   ╘══════════════════════════════════════════════════════════════╛
   

   You are now ready to upgrade SevOne Data Insight using the Graphical User Interface Installer.

   Important: Remove packages
   You may have expect and tcl in your environment. Prior to starting the upgrade, please execute the following command to remove these packages.

   
   $ sudo yum remove -y expect tcl
   

2. Using a web browser of your choice, enter the URL returned above. For example, https://10.49.14.243:3000.
   
   Note: You will also need the credentials (Username and Password) returned. These credentials are also stored in /etc/sevone-guii/creds file.

   Example

   
   $ cat /etc/sevone-guii/creds | jq
   {
    "password": "}_G<{oJ|`!",
    "tokenSecret": "DxiZlUNDqBiTrWWseHvRYIgaLQOGISTs",
    "username": "admin"
   }
   

3. Click Update Cluster to update SevOne Data Insight. For example, upgrade from SevOne Data Insight 6.7.0+160 to SevOne Data Insight 6.8.0+<latest version>.
4. Enter the credentials returned to perform the Self-Service Upgrade. For example, Username: admin and Password: }_G<{oJ|`!
   
   Note:
   To use the Graphical User Interface installer in dark theme, click next to SevOne logo.
   For help on what each upgrade step does, click button in the upper-right corner.
   Important: All the screenshots below are based on the example being used to write this document. Your total number of tasks passed (ok), skipped, failed, ignored, unreachable, or unexecuted will vary based on your setup. The tasks failed must be addressed as ansible has not ignored them.
5. The graphical user interface installer checks the Current Version and the Upgrade Available version. If the current version is older than the upgrade version available, you are now ready to continue to Pre-Upgrade.
   
   Important: Example
   Current Version is on SevOne Data Insight 6.7.0+160.
   Upgrade Available version is SevOne Data Insight 6.8.0+40 (for this example).
   
   Important: During the Self-Service Upgrade, if you experience network connectivity issue or the upgrade has been halted for any reason, the self-service upgrade will resume from the step where it left off after the issue is resolved. However, if you are at the Upgrade step and the self-service upgrade has been halted for any reason, self-service upgrade will show a message requesting you to contact SevOne Support.
   To resume with the Self-Service Upgrade, using a web browser of your choice, re-enter the URL the setup script has returned. For example, https://10.49.14.243:3000.
6. Click Continue to Pre-Upgrade>button.
7. You are now ready to run the pre-upgrade. Click Run Pre-Upgrade button.
   Pre-Upgrade step runs various checks to ensure that SevOne Data Insight cluster is healthy before the upgrade. It takes backups of the database and creates a copy of the security keys.
   To view the logs for a task, click for the task you need the details for. The pop-up has Copy to clipboard button which allows you to copy all the contents in the pop-up and paste it into a file.
8. Click Continue button to upgrade SOA.
   
   Note: SOA version
   SOA must be on the latest version on all appliances in SevOne NMS cluster. Command Line Interface (CLI) must be used to upgrade SOA on all peers as the graphical user interface (GUI) only upgrades SOA for the NMS appliance you are connected to.
   Add flag --all-peers if you want to install SOA on all peers in the cluster.

   
   $ sevone-cli soa upgrade \
   /opt/SevOne/upgrade/utilities/SevOne-soa-*.rpm \
   <enter SevOne NMS IP address> --all-peers
   

9. You are now ready to upgrade SOA. Click Run Upgrade SOA button. This can take a few minutes to run.
   
10. Click Continue button to configure Data Insight chart values, if required. This allows you to add ingress settings, or tune defaults for large deployments, in addition to verifying previously configured values. If you do not want to configure the chart values i.e., modify the ingress or GraphQL or printer or MySQL settings, click the Continue button to go to the next step to perform the upgrade.
    
    1. From Configuration drop-down, choose a configuration file from the list. For example, the primary configuration file is di_custom_guii.yaml. If SevOne Data Insight has already been configured, you can select that configuration from the drop-down list to edit it. If multiple configuration files are present, the files will be applied in the alphabetical order.
       
    2. Ingress
       1. Base Domain - a base domain is added to the end of every host. For example, .reporting.
       2. Base Path - allows SevOne Data Insight to host itself at an arbitrary subpath. This is useful to allow for some types and routing, or to have SevOne Data Insight act as a page of a larger whole.
          
          Important: This can be a regular expression. If it includes a capture group, you MUST change captureGroup as well.
       3. Capture Group - indicates which part of the path gets passed through nginx. By default, nginx passes everything after the first /. But, if you add a regex in the Base Path, Capture Group must be changed.
       4. Hosts - Click to add the host name(s) that SevOne Data Insight must respond to.
       5. TLS Secret Name - name of kubernetes secret that contains the TLS certificate and key. Please set this to datainsightcerts.
    3. GraphQL
       1. Replicas - contains the number of GraphQL pods to run. Increase this number if you have multiple agent nodes and several concurrent users. The maximum value allowed is 10.
       2. Environmental Variables
          • Auth Timeout - duration in seconds before an idle user times out. The default value is 3600. The maximum value allowed is 100,000.
          • Skip Report Migration Dry Run - enable check box if you want to skip the report migration dry run.
          • Skip Check Datasources - enable check box if you want to skip the step to check datasources.
    4. Printer
       1. Replicas - contains the number of printer pods to run. Increase this number for large amounts of scheduled reports.
    5. MySQL
       1. Primary
          • Extra Flags - additional flags to pass to MySQL when starting on the primary instance. Only maximum connections must be passed. Use example for exact formatting. For example, --max_connections=1000.
    6. Maps
       
       Warning: IMPORTANT: Missing / misconfigured tile server for Map mode
       The tile server for Map (geospatial) mode is missing or misconfigured in SevOne Data Insight 3.13.0 - 3.13.9, 6.5.0 - 6.5.4, and 6.6.0. You must successfully perform an upgrade to SevOne Data Insight 6.7.0 or above and at this step, you are required to configure variables Tileserver and Attribution to obtain the proper tiles for the maps.

       Note: Failure to set variable Tileserver while performing an upgrade from SevOne Data Insight 3.13.0 - 3.13.9, 6.5.0 - 6.5.4, and 6.6.0 to SevOne Data Insight 6.7.0 or above, will convert your current map in Maps mode to Canvas mode without the ability to switch back to Map mode until the tile server is provided.
       If the owner of the selected tile server requires to be given credit for the tiles being used, set variable Attribution. Text entered in variable Attribution will appear in the bottom right corner of each map.

       
       For further assistance, please contact SevOne Support Team or your Technical Account Manager to configure this properly.
       1. Tileserver - enter the tile server URL containing the token. For details on how to create a tile server URL, please refer to [SevOne Data Insight Administration Guide](https://www.ibm.com/docs/en/sevone-npm/6.8?topic=operations-sevone-data-insight-administration-guide) < Configuration < Helm Chart < Configure Maps < section Create Tile Server URL.
          • for Dark mode, https://api.mapbox.com/styles/v1/sevone/cllfjv99c01mp01ql9neoau07/tiles/256/{z}/{x}/{y}@2x?access_token=<ACCESS_TOKEN>
          • for Light mode, https://api.mapbox.com/styles/v1/sevone/cllfl73f101qb01qnc7wh22i5/tiles/256/{z}/{x}/{y}@2x?access_token=<ACCESS_TOKEN>
       2. Attribution - enter some attribution; props to stamen for providing tiles, etc.
11. Click Save. Configuration is saved in /opt/SevOne/chartconfs/di_custom_guii.yaml.
    
    Important: Once the configuration is saved, click Continue button to upgrade.
12. You are now at the Upgrade step.
    
    
13. Click Run Upgrade button to run the upgrade.
    
    Important: Allows you to start the upgrade and show the progress.
    It will upgrade your operating system from CentOS 7 to RHEL 8 and can take over an hour to run.
    
    
    SevOne Data Insight upgrade has now completed.
    Note: The task list can be long. Search tasks capability is available to search for text you are looking for in the task list.
    Note: To view the logs for a task, click for the task you need the details for. The pop-up has Copy to clipboard button which allows you to copy all the contents in the pop-up and paste it into a file.
14. Click the Continue button.
    
    Important: This indicates that the upgrade has completed successfully.
15. You may now link the link Click Here.
16. Link Click Here will launch SevOne Data Insight. You will notice that SevOne Data Insight is now on the upgraded version i.e., 6.8.0.
    

Example: Upgrade not required
If you are already on the version you are trying to upgrade to, you will get the You can proceed with redeploy/install message. This allows you to upgrade to a later version, if available.

Important: Additional details on GUI Installer can be found in SevOne Data Insight Administration Guide > Tools > section GUI Installer.

Note: Please refer to SevOne Data Insight Administration Guide for configuration and deployment details and SevOne Data Insight Troubleshooting / Debug / Tools Guide for troubleshooting scenarios, how to debug SevOne Data Insight, and various tools available.