SD-WAN Versa Collector Upgrade Process Guide

SevOne Documentation

All documentation is available from the IBM SevOne Support customer portal.

© Copyright International Business Machines Corporation 2023.

All right, title, and interest in and to the software and documentation are and shall remain the exclusive property of IBM and its respective licensors. No part of this document may be reproduced by any means nor modified, decompiled, disassembled, published or distributed, in whole or in part, or translated to any electronic medium or other means without the written consent of IBM.

IN NO EVENT SHALL IBM, ITS SUPPLIERS, NOR ITS LICENSORS BE L3IABLE FOR ANY DAMAGES, WHETHER ARISING IN TORT, CONTRACT OR ANY OTHER LEGAL THEORY EVEN IF IBM HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES, AND IBM DISCLAIMS ALL WARRANTIES, CONDITIONS OR OTHER TERMS, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, ON SOFTWARE AND DOCUMENTATION FURNISHED HEREUNDER INCLUDING WITHOUT LIMITATION THE WARRANTIES OF DESIGN, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT.

IBM, the IBM logo, and SevOne are trademarks or registered trademarks of International Business Machines Corporation, in the United States and/or other countries. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on ibm.com/trademark.

About

This document provides details on how to do the upgrade of SD-WAN Versa collector.

Please do not run sevone-cli command from a subdirectory under /opt/SevOne/upgrade and /var/log/pods. It can be run from any directory except for from subdirectories under /opt/SevOne/upgrade and /var/log/pods.
Terminology usage...
In this guide if there is,
  • [any reference to master] OR
  • [[if a CLI command (for NMS or Kubernetes or Redis) contains master] AND/OR
  •  [its output contains master]],
   it means leader or control plane.

And, if there is any reference to slave or worker, it means follower or agent.

Upgrade Versa Collector using Graphical User Interface

You must be on SD-WAN >= 2.9 to perform an upgrade using GUI.

This section describes how to upgrade SD-WAN Versa Collector from 6.5 to 6.6 using steps based on the GUI.

  1. Using ssh, log into SD-WAN Versa collector control plane node as sevone.

    $ ssh sevone@<SD-WAN collector 'control plane' node IP address or hostname>
    

    Example: Currently on SD-WAN Versa Collector 6.5 and to be upgraded to SD-WAN Versa Collector 6.6

    $ ssh sevone@10.128.9.13
    
  2. Change directory to /opt/SevOne/upgrade.

    $ cd /opt/SevOne/upgrade
    
  3. Download the following (latest) files from IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact IBM SevOne Support for the latest files. You must place these files in /opt/SevOne/upgrade directory.

    1. sevone_solutions_sdwan_versa-v6.6.0-build.35.tgz

    2. sevone_solutions_sdwan_versa-v6.6.0-build.35.tgz.sha256.txt

    3. signature-tools- -build.<###>.tgz
      For example, signature-tools-2.0.1-build.1.tgz

    4. signature-tools- -build.<###>.tgz.sha256.txt
      For example, _signature-tools-2.0.1-build.1.tgz.sha256.txt

      $ ls -lah /opt/SevOne/upgrade
      

      You should see all the files mentioned above in this directory.

      /opt/SevOne/upgrade/freight directory contains yum packages for air-gapped (no internet) environments.
  4. Stop / reset the kubernetes.

    $ sevone-cli cluster down
    
  5. Execute the following commands to verify the checksum of the code signing tool before extracting it.

    $ (cd /opt/SevOne/upgrade && cat $(ls -Art signature-tools-*.tgz.sha256.txt | \
    tail -n 1) | sha256sum --check)
    
    $ sudo tar xvfz $(ls -Art /opt/SevOne/upgrade/signature-tools-*.tgz | \
    tail -n 1) -C /
    
  6. Verify the signature of Solutions .tgz files.

    $ /usr/local/sbin/SevOne-validate-image \
    -i $(ls -Art /opt/SevOne/upgrade/sevone_*.tgz | tail -n 1) \
    -s $(ls -Art /opt/SevOne/upgrade/sevone_*.tgz.sha256.txt | tail -n 1)
    

Upgrade

Execute the following steps to upgrade SD-WAN Versa Collector using Graphical User Interface.

To upgrade SD-WAN, you need to install sevone-cli, if not installed already.
  1. Navigate to /opt/SevOne/upgrade directory.

    $ cd /opt/SevOne/upgrade
    
  2. Remove directory /opt/SevOne/upgrade/utilities.

    $ rm -rf /opt/SevOne/upgrade/utilities
    
  3. Extract the latest build.

    $ tar xvfz $(ls -Art /opt/SevOne/upgrade/sevone_*.tgz | \
    tail -n 1) -C /opt/SevOne/upgrade/ ./utilities
    
  4. Check if sevone-cli is installed already.

    Example

    $ rpm -qa |grep sevone-cli
    sevone-cli-2.4.0-3.el7.x86_64
    

    The command above returning a version indicates that sevone-cli is installed.

    If sevone-cli is not installed, the command above will not return any version. And, you will need to execute the following command to install sevone-cli.

    $ sudo rpm -Uvh /opt/SevOne/upgrade/utilities/sevone-cli-*.rpm
    
  5. Change directory to /etc/ansible/group_vars/all . Using a text editor of your choice, add the following variables in solutions.yaml file.

    Example: solutions.yaml file for Versa Collector

    $ vi /etc/ansible/group_vars/all/solutions.yaml
    
    app_solutions:
    chart: solutions-sdwan-versa
    enabled: true
    name: solutions-sdwan-versa
    namespace: default
    container_log_max_files: 30
    container_log_max_size: 200Mi
    k3s_config_extra: 'kube-apiserver-arg: service-node-port-range=9900-33000'
    
    It is mandatory to add these variables in solutions.yaml file.
    1. container_log_max_size - Set the maximum size of container log file before it is rotated. Recommended value is 200Mi.
    2. container_log_max_files - Set the maximum number of container log files that can be present for a container. Recommended value is 30.
  6. Change directory to /opt/SevOne and create .collector_installed file using the following command.

    It is mandatory to create this file to skip PAS sizing checks during pre-check.
    $ touch /opt/SevOne/.collector_installed
    
Before doing an upgrade, please do the following.
  1. Upgrade SOA manually using the steps available in SD-WAN Versa Collector Deployment / Configuration Guide > section Installation > subsection using Command Line Interface > subsection Upgrade SOA.
  2. If you have used GUI for upgrading the collector in the past, please remove the existing SSU before starting an upgrade by executing the following command.
    $ sudo rpm -e sevone-guii
    

using Command Line Interface

Execute the following command to update SOA and upgrade SD-WAN Versa Collector.

When running the upgrade using the Command Line Interface, it validates the checksum, extracts the upgrade file, upgrades sevone-cli (if required), upgrades SOA, attempts to read NMS IP list from SD-WAN Versa Collector database, imports TopN views in SevOne NMS & OOTB reports in SevOne Data Insight and configures the SSH keys for the NMS being used as a datasource.

After the upgrade completes successfully, SevOne highly recommends a reboot to update the kernel as it does not update automatically. To reboot, execute command, sudo reboot.

When you run the command below, you will be prompted to enter the password two times. At the first prompt, enter the password you would enter when you SSH to SD-WAN Versa collector control plane node as sevone. At the second prompt, enter the password you would enter when you SSH to SevOne NMS as root.

Update SD-WAN Versa Collector without using the Graphical User Interface Installer

$ /usr/local/sbin/SevOne-validate-image \
-i $(ls -Art /opt/SevOne/upgrade/sevone_solutions_sdwan_versa-*.tgz | tail -n 1) \
-s $(ls -Art /opt/SevOne/upgrade/sevone_solutions_sdwan_versa-*.tgz.sha256.txt | tail -n 1) \
--installer solutions  --installer-opts '--no_screen'
If you do not want to upgrade SOA, please perform the following steps.
  1. Create SevOne NMS SSH keys.

    $ sevone-cli soa setup_keys
    
  2. Run the following command to start the upgrade process.

    Upgrade using Command Line Interface

    $ /usr/local/sbin/SevOne-validate-image \
    -i $(ls -Art /opt/SevOne/upgrade/sevone_solutions_sdwan_versa-*.tgz | tail -n 1) \
    -s $(ls -Art /opt/SevOne/upgrade/sevone_solutions_sdwan_versa-*.tgz.sha256.txt | tail -n 1) \
    --installer solutions  --installer-opts '--skip-soa --no_screen'
    

using Graphical User Interface

  1. The setup script in the command below updates SOA on all your datasources and configures the Graphical User Interface Installer. This steps takes about 1-2 minutes to run.

    $ /usr/local/sbin/SevOne-validate-image \
    -i $(ls -Art /opt/SevOne/upgrade/sevone_solutions_sdwan_versa-*.tgz | tail -n 1) \
    -s $(ls -Art /opt/SevOne/upgrade/sevone_solutions_sdwan_versa-*.tgz.sha256.txt | tail -n 1) \
    --installer solutions-gui
    

    Example: The command returns the following

    ╒═══════════════════════════════════════════════════════════════╕
    │ SEVONE GUI INSTALLER                                          │
    ╞═══════════════════════════════════════════════════════════════╡
    │ Please open https://10.128.9.13:3000 in your web browser to   │
    │ access the GUI Installer.                                     │
    ├───────────────────────────────────────────────────────────────┤
    │ Your credentials are:                                         │
    │ - Username: admin                                             │
    │ - Password: @r~.+@K?Kh                                        │
    ├───────────────────────────────────────────────────────────────┤
    │ If you ever lose your credentials, they are stored in:        │ 
    │ /etc/sevone-guii/creds                                        │
    ╘═══════════════════════════════════════════════════════════════╛
    

    You are now ready to upgrade using the Graphical User Interface Installer.

  2. Using a web browser of your choice, enter the URL the setup script has returned. For example, https://10.128.9.13:3000.

    Versa GUI Installer Start
    You will also need the credentials (Username and Password) that the setup script returns. These credentials are also stored in /etc/sevone-guii/creds file.

    Example

    $ cat /etc/sevone-guii/creds | jq  
    
    {
        "password": "@r~.+@K?Kh",
        "tokenSecret": "KRfzXkEGHPqRMgbibTxkSvHCMMFhoYzt",
        "username": "admin"
    }
    
  3. Click Update Cluster to update SD-WAN Versa Collector. For example, upgrade from SD-WAN Versa Collector 6.5.0+53 to SD-WAN Versa Collector 6.6.0+25.

  4. Enter the credentials returned to perform the Self-Service Upgrade. For example, Username: admin and Password: @r~.+@K?Kh

    Versa GUI Installer Credential
    To use the Graphical User Interface installer in dark theme, click GUI Install Dark Theme next to SevOne logo.

    For help on what each upgrade step does, click GUI Installer Help Icon button in the upper-right corner.
    IMPORTANT
    All the screenshots below are based on the example being used to write this document. Your total number of tasks passed (ok) , skipped, failed, ignored, unreachable, or unexecuted will vary based on your setup. The tasks failed must be addressed as ansible has not ignored them.
  5. The graphical user interface installer checks the Current Version and the Upgrade Available version. If the current version is older than the upgrade version available, you are now ready to continue to Configure.

    Versa GUI Installer Check Versions
    Example
    Current Version is on SD-WAN Versa Collector 6.5.0+53.
    Upgrade Available version is SD-WAN Versa Collector 6.6.0+25.
    During the Self-Service Upgrade, if you experience network connectivity issue or the upgrade has been halted for any reason, the self-service upgrade will resume from the step where it left off after the issue is resolved. However, if you are at the Deploy step and the self-service upgrade has been halted for any reason, self-service upgrade will show a message requesting you to contact IBM SevOne Support.

    To resume with the Self-Service Upgrade, using a web browser of your choice, re-enter the URL the setup script has returned. For example, https://10.128.9.13:3000.
  6. Click the Continue to Configure button to configure SD-WAN Versa chart values, if required.

    If you do not want to configure the chart values, click Continue button to upgrade SOA.
    Limitation applies with GUI Installer version 2.0.0 only
    In case of multi-tenancy, the GUI Installer only edits the first tenant. Any additional tenant configuration must be performed manually. Using a text editor of your choice, edit /opt/SevOne/chartconfs/solutions-sdwan-versa-<tenant #>_custom-guii.yaml to set your configuration settings. This is the only scenario where the .yaml file needs to be edited manually.
    Versa GUI Installer Configure
    1. From Configuration drop-down, choose a configuration file from the list. For example, the primary configuration file is solutions-sdwan-versa_custom_guii.yaml. If SD-WAN Versa collector has already been configured, you can select that configuration from the drop-down list to edit it. If multiple configuration files are present for the same tenant, the files will be applied in the alphabetical order.

      Versa GUI Installer Configure Primary
    2. Collector Service

      • Credentials (All values must be base64-encoded format)
        • Controller Credentials
          • Username - The username for Versa Director credentials with admin-level read privilege.
          • Password - The password for Versa Director.
        • NMS Credentials
          • NMS API Credentials
            • Username - The SevOne NMS user name for an administrator-level account.
            • Password - The SevOne NMS password.
          • SSH Credentials
            • Username - The SevOne NMS user name for ssh access to the appliance. It is recommend to set to root in base64-encoded format.
            • Password - The SevOne NMS password for root user.
        • DI Credentials
          • DI API Credentials
            • Username - The SevOne Data Insight user name for an adminstrator-level account.
            • Password - The SevOne Data Insight password.
        • Syslog Receiver Port - The port on which the collector listens for non-flow syslog data sent by Versa Analytics.
    3. Collector Configuration

      • MSP Name - The Managed Service Provider (MSP) name for this instance. MSP is a grouping of one or more tenants.
    4. Log

      • Log Level - Defines the log-level for the collector. Value can be info, debug, warning, or error.
    5. Jaeger

      • Disabled - Select the check box to disable Jaeger tracing.
    6. Load Reports

      • Disabled - Set the check box to not import TopN views and OOTB reports.
    7. Vendor Controller Settings

      • Versa Director Settings
        • Versa Director API URL - The API URL of Versa Director.
        • Insecure TLS - Select the check box to enable insecure TLS connection by skipping certification verification. This is necessary for servers with self-signed server certificates.
    8. NMS

      • NMS API Settings
        • NMS API IP / Hostname - The hostname or IP address for SOA and REST API endpoints. i.e., targeted SevOne NMS.
        • Insecure TLS - Select the check box to enable insecure TLS connection by skipping certification verification. This is necessary for servers with self-signed server certificates.
      • DI
        • DI API Settings - The hostname or IP address for targeted SevOne Data Insight.
        • Insecure TLS - Select the check box to enable insecure TLS connection by skipping certification verification. This is necessary for servers with self-signed server certificates.
        • Tenant - Tenant name for this SevOne Data Insight instance. This is an internal name used to keep settings and cached data segregated by tenant. Default value is SevOne.
    9. Flow Augmentor Service Settings

      • Enable - Select the check box to enable Flow Augmentor installation.
      • Flow Receiver Port - The port on which Flow Augmentor listens for inbound flows. The port number can range from 9000 - 33000.
    10. Flow Augmentor Configuration
      * Flow Augmentor Sender Configuration

      • Flow Augmentor Sender Buffer Size - Sender output buffer size in number of packets.
      • DNC IP - IP address of the DNC, where the augmented flows are sent.
      • Port No - Port of DNC, where the the augmented flows are sent.
  7. Click Save. Configuration is saved in /opt/SevOne/chartconfs/solutions-sdwan-versa_custom_guii.yaml.

    Once the configuration is saved, click Continue button to upgrade SOA.
  8. Click Continue button to Upgrade SOA.

    SOA version
    SOA must be on the latest version on all appliances in SevOne NMS cluster. Command Line Interface (CLI) must be used to upgrade SOA on all peers as the graphical user interface (GUI) only upgrades SOA for the NMS appliance you are connected to.
    Add flag --all-peers if you want to install SOA on all peers in the cluster.
    $ sevone-cli soa upgrade \
    /opt/SevOne/upgrade/utilities/SevOne-soa-*.rpm \
    <enter SevOne NMS IP address> --all-peers
    
    Versa GUI Installer Upgrade SOA
  9. You are now ready to upgrade SOA. Click Run Upgrade SOA button. This can take a few minutes to run.

    Versa GUI Installer SOA Upgraded
  10. Click the Continue button to Pre-Check.

    Pre-Check step runs various checks to ensure that SD-WAN Versa collector cluster is healthy before the deployment. It takes backups of the database and creates a copy of the security keys.
    Versa GUI Installer Run PreCheck
  11. You are now ready to run the pre-check. Click the Run Pre-Check button.

    Versa GUI Installer Pre Check Completed
    To view the logs for a task, click GUI Installer Eye Icon for the task you need the details for. The pop-up has Copy to clipboard button which allows you to copy all the contents in the pop-up and paste it into a file.
  12. Click the Continue button to Deploy.

    Versa GUI Installer Run Deploy
  13. Click the Run Deploy button to deploy the collector. This can take a few minutes to run.

    Versa GUI Installer Deploy Completed
    To view the logs for a task, click GUI Installer Eye Icon for the task you need the details for. The pop-up has Copy to clipboard button which allows you to copy all the contents in the pop-up and paste it into a file.
  14. Click the Continue button to Post-Check.

    Please skip Post-Check if you do not want to overwrite the existing flow view. Also, please verify the status of pods. For more details, please refer to SD-WAN Versa Collector Deployment / Configuration Guide > Verification > section Check Pods.
    Versa GUI Installer Run Post Check
  15. Click the Run Post-Check button to run the post-check. This can take a few minutes to run.

    Versa GUI Installer Post Check Completed

    SD-WAN Viptela Collector upgrade has now completed.

    The task list can be long. Search tasks capability is available to search for text you are looking for in the task list.
    To view the logs for a task, click GUI Installer Eye Icon for the task you need the details for. The pop-up has Copy to clipboard button which allows you to copy all the contents in the pop-up and paste it into a file.
  16. Click the Continue button.

    Versa GUI Installer Upgrade Finished
    This indicates that the upgrade has completed successfully.
    If there was a kernel upgrade for this release, it is recommended to reboot the cluster after the upgrade to install the right kernel version.
  17. You may now click the link Click Here.

  18. Link Click Here will launch SevOne Data Insight, where you can view the data collected by the collector.

    Versa GUI Installer Launch After Upgrade
Example: Upgrade not required
If you are already on the version you are trying to upgrade to, you will get the You can proceed with redeploy/install message. This allows you to upgrade to a later version, if available.
Versa GUI Installer Redeploy Or Install

FAQs

Change Ports

  1. For details on default port numbers and its respective config file locations for the client and the API, please refer to SevOne NMS Port Number Requirements Guide > section Solutions Deployment > for SD-WAN Versa collector.

  2. The following table lists the default port numbers and its respective config file location for the client and API.

    Name Default Port Config File Location
    Client 3000 /etc/sevone-guii/client.custom.yaml
    API 3001 /etc/sevone-guii/api.custom.yaml
  3. If you need to change the port number, using a text editor of your choice, edit the .yaml file to change the setting and save the file.

    Example: Change port number for Client

    $ vi /etc/sevone-guii/client.custom.yaml
    main:
      port: 3000  # <-- change port number
    

    Example: Change port number for API

    $ vi /etc/sevone-guii/api.custom.yaml
    main:
      port: 3001  # <-- change port number
    
  4. Restart the client and API services.

    In case of any configuration changes, you need to restart both client and API services.
    $ sudo systemctl restart sevone-guii-@api
    
    $ sudo systemctl restart sevone-guii-@client
    

Manage Services

The graphical user interface installer services can be started / stopped using the standard systemd commands.

$ sudo systemctl status sevone-guii-@api

$ sudo systemctl status sevone-guii-@client
 
$ sudo systemctl start sevone-guii-@api
 
$ sudo systemctl start sevone-guii-@client
 
$ sudo systemctl stop sevone-guii-@api
 
$ sudo systemctl stop sevone-guii-@client

View Logs

The logs can be viewed using journalctl.

$ journalctl -u sevone-guii-@api [-f]
 
$ journalctl -u sevone-guii-@client [-f]

Clear ARA Status

When running incremental upgrades, ARA (Ansible Run Analysis) status from the previous upgrade must be cleared. Execute the following command to clear the ARA status.

$ rm /etc/sevone-guii/ara/server/ansible.sqlite
  
$ sudo systemctl restart ara-server