SevOne SDN Collector Upgrade Process Guide

SevOne Documentation

All documentation is available from the IBM SevOne Support customer portal.

© Copyright International Business Machines Corporation 2023.

All right, title, and interest in and to the software and documentation are and shall remain the exclusive property of IBM and its respective licensors. No part of this document may be reproduced by any means nor modified, decompiled, disassembled, published or distributed, in whole or in part, or translated to any electronic medium or other means without the written consent of IBM.

IN NO EVENT SHALL IBM, ITS SUPPLIERS, NOR ITS LICENSORS BE LIABLE FOR ANY DAMAGES, WHETHER ARISING IN TORT, CONTRACT OR ANY OTHER LEGAL THEORY EVEN IF IBM HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES, AND IBM DISCLAIMS ALL WARRANTIES, CONDITIONS OR OTHER TERMS, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, ON SOFTWARE AND DOCUMENTATION FURNISHED HEREUNDER INCLUDING WITHOUT LIMITATION THE WARRANTIES OF DESIGN, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT.

IBM, the IBM logo, and SevOne are trademarks or registered trademarks of International Business Machines Corporation, in the United States and/or other countries. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on ibm.com/trademark.

About

NOTICE
This document describes the process to upgrade SevOne SDN Collector 2.7 Solution running either on a SevOne NMS appliance or a non-SevOne NMS appliance host to SevOne SDN Collector 6.6 Solution.

If the upgrade is executed on SevOne NMS appliance then, it is mandatory to execute the steps on both the leader and the HSA appliances.
Terminology usage...
In this guide if there is,
  • [any reference to master] OR
  • [[if a CLI command (for NMS or Kubernetes or Redis) contains master] AND/OR
  •  [its output contains master]],
   it means leader.

And, if there is any reference to slave, it means follower.

Prerequisites

General

The tar file cisco-aci-agent-solution-v6.6.0-build-235714.tar.gz can be downloaded from IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact IBM SevOne Support for the tar file. The tar must contain the following components:

  • Cisco ACI Agent Docker image
  • launch script
  • conf directory
  • version.txt
  • install.sh

Docker

Docker 18.06.1.ce is a minimum requirement. If docker is not installed and you need to install Docker components, please contact IBM SevOne Support.
  • SSH into SevOne NMS or the system on which the collector needs to be deployed.
  • To avoid typing sudo whenever you run the docker command, add your username to the docker group.
  • For this to take effect, you will need to log out and log back in.
    $ usermod -aG docker <username>
    

    $ usermod -aG docker sevone
    

Upgrade: Forward Migration

The process below applies only to upgrade SevOne SDN Collector 2.7 Solution to SevOne SDN Collector 6.6 Solution.
Reverse Migration from SevOne SDN Collector 6.6 Solution to SevOne SDN Collector 2.7 Solution is not tested and supported.
  1. SSH into SevOne NMS or the system on which the collector needs to be deployed .

    $ ssh root@<NMS appliance>
    
  2. Download the following (latest) files from IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact IBM SevOne Support for the latest files. You must place these files in /root directory of SevOne NMS appliance.

    1. cisco-aci-agent-solution-v6.6.0-build-235714.tar.gz
    2. cisco-aci-agent-solution-v6.6.0-build-235714.tar.gz.sha256.txt
    3. signature-tools- -build.<###>.tgz
      For example, signature-tools-2.0.1-build.1.tgz
    4. signature-tools- -build.<###>.tgz.sha256.txt
      For example, signature-tools-2.0.1-build.1.tgz.sha256.txt
  3. Execute the following commands to verify the checksum of the code signing tool before extracting it.

    $ (cat $(ls -Art signature-tools-*.tgz.sha256.txt | \
    tail -n 1) | sha256sum --check)
    
    $ sudo tar xvfz $(ls -Art signature-tools-*.tgz | \
    tail -n 1) -C /
    
  4. Extract the latest build.

    $ tar xvf cisco-aci-agent-solution-v6.6.0-build-235714.tar.gz
    
  5. Perform the pre-check of your environment and monitor the output. Ensure that there are no failures reported in the output.

    $ SevOne-act check tables
    
    The pre-check must complete successfully before you can continue to the next step. If the pre-check does not complete successfully, please resolve the issue(s) before continuing or contact SevOne Support.

Install on SevOne NMS Appliance

The steps below must be performed on both the leader and HSA appliance in a Cluster environment.

Execute the following command if the installation is on SevOne NMS appliance.

$ ./install.sh --upgrade

  1. In SevOne Cisco SDN Collector 6.6 Solution, the install script operates on the settings.env and multi_site_config.json files. The user may choose to keep these files in a specific directory and provide the directory path as an argument to the install script's --conf-path option, as demonstrated below:
    $ ./install.sh --upgrade --conf-path <directory path for settings.env/multi_site_config.json files>
    
    The --conf-path argument must be an absolute path. If the settings files in /opt/cisco-aci-agent/conf directory are used, the user may choose to not provide the directory path in --conf-path argument.
  2. If your SevOne NMS is SSL verified, provide the CA bundle to the install script using the --ssl-verify option, as demonstrated below:
    $ ./install.sh --upgrade --ssl-verify <path-to-CA-bundle>
    
    The --ssl-verify argument must be an absolute path. If this option is used, please make sure the following variables are set in the settings.env file.
    • SEVONE_API_SECURITY=True
    • SEVONE_API_COLLECTION_PORT=443
    • SEVONE_API_CONFIG_PORT=443

It will perform the following tasks.

  1. Enables and starts Docker services, if required

  2. Sets up the pre requisite directories and files

  3. Validates the checksum file

  4. Extracts the cisco-aci-agent-solution-v6.6.0-build-235714.tar.gz file

  5. Loads the Docker image

  6. Adds execute permissions to the launcher script

  7. Extracts a sample certificate-private key pair that can be used for Signature-based Authentication.

    Do not use the sample keys as they are the same for all customers. You must generate your own signature pair.
  8. Executes the installer to setup the settings.env and multi_site_config.json files. This option can also be executed independently by the user using the launch script. For detailed instructions, please refer to SevOne SDN Collector Set Collector Using 'launch configure' Option.

    To modify the fault filters based on the latest filters available for SDN 6.6, please refer to SevOne SDN Collector Filter Alerts Using 'launch generate-fault-config' Option.
  9. Extracts and handles the dependencies, required for ACI. This includes installing the autossh package, creating a systemD service, modifying sshd 's settings, restarting the sshd service and starting the autossh tunnel forwarding service . The step was created as a shortcut when installing on SevOne NMS appliances however, it will not work when the installation is not on SevOne NMS.

    1. In a pop-up editor, you will see contents similar to the sample below.

      # TARGET=apic # this is the name
      # LOCAL_ADDR=0.0.0.0 # which local interface to bind to <all>
      # LOCAL_PORT=9999 # which local port to use
      # APIC_ADDR=172.21.100.250 # where is the apic
      # APIC_PORT=443 # what port is the API on the APIC
      
      TARGET=apic
      LOCAL_ADDR=0.0.0.0
      LOCAL_PORT=9999
      APIC_ADDR=172.21.100.250
      APIC_PORT=443
      
    2. Press Ctrl+x . However, if you attempt to install via WebEx, Ctrl+x is not possible from the WebEx session. You will need a local user to perform this step for you.

  10. Refreshes the cache so that any device-ids maintained in a cache on the machine may be deleted.

  11. Adds objects and indicators to SevOne NMS. This includes setting the topology constraints and create Object Types, Indicator Types, Device Groups and rules, and Metadata Schema.

  12. Sets up logrotate.

Import OOTB Reports

To import out-of-the-box (OOTB) reports, please execute the following command.

The command below to import OOTB reports, is not required to be run on an HSA.

Execute this command to import OOTB reports

$ ./install.sh --import-ootb-reports
OOTB reports will be imported to your SevOne NMS only when the --import-ootb-reports flag is enabled.

Install on non-SevOne NMS Appliance

Execute the following command if the installation is not on SevOne NMS appliance.

$ ./install.sh --configure --non_pas_device

It will perform the tasks (3.) through (7.) and (9.) through (11.) as described above.

Configure the 'cron' file for Data Collection

Perform the following actions to configure the cron file for data collection. When running on a SevOne NMS cluster, please execute the crontab commands below on both the leader and HSA so that failover can be handled.

  1. Execute the following command.

    $ crontab -e
    
  2. Add the following line to start the collection agent.

    */5 * * * * /opt/cisco-aci-agent/launch run
    

    1. If the user has stored the settings.env and multi_site_config.json files in a directory other than the default (/opt/cisco-aci-agent/conf), the directory path must be provided as an argument to the launch script's --conf-path option in the crontab, as demonstrated below.
      */5 * * * * /opt/cisco-aci-agent/launch run --conf-path <directory path for settings.env/multi_site_config.json files>
      
      The --conf-path must be an absolute path.
    2. If your SevOne NMS is SSL verified, provide the CA bundle to the install script using the --ssl-verify option, as demonstrated below.
      */5 * * * * /opt/cisco-aci-agent/launch run --ssl-verify <path-to-CA-bundle>
      
      The --ssl-verify must be an absolute path. Also, if you see this option, please make sure to set the following variables in the settings.env file.
      • SEVONE_API_SECURITY=True
      • SEVONE_API_COLLECTION_PORT=443
      • SEVONE_API_CONFIG_PORT=443
    You may use this for both the leader and the HSA appliance crontab entry. It checks the /SevOne.masterslave.master file for its value to determine whether it needs to run or not.
  3. Save and close the file.

    After installing the cron, it might take a few minutes before it executes the job. You may monitor the cron logs using the following command.
    $ tail -f  /var/log/cron  | grep launch
    

    Once a log appears as a result of this command, press Ctrl+C to exit. You may then proceed to verify that data appears in SevOne NMS.

IMPORTANT
After the upgrade, if you have SevOne Data Insight, please refer to SevOne SDN Collector Solution Widgets for Data Insight Installation Guide to obtain details on how to update the SDN widgets.

Troubleshooting / Useful Commands

Obtain List of Running Docker Containers

$ docker ps

CONTAINER ID   IMAGE                                            COMMAND                  CREATED       STATUS      PORTS   NAMES
ecf5881e406d   docker.sevone.com/sc/collectors/cisco-aci:v6.6.0 "python source/main.…"   5 days ago    Up 5 days           cisco-aci-agent-2022-06-22-10-15-25

View Log

To confirm that the collector is running and checking for errors, tail the log file you have setup. The log file will be available in the location specified in the multi_site_config.json file as cisco_log_filepath setting.

$ tail -f <data-log-file-name>.log

Reset settings.env and multi_site_config.json

In order to reset the settings.env and multi_site_config.json files to its original factory settings, the launch script has been provided with a reset-settings option. The reset-settings option, when executed, it will reset the settings.env and multi_site_config.json files.

Please use reset-settings option with caution as the settings.env and multi_site_config.json files will be overwritten. Please make sure that the files are backed-up before running this option.

Example

$ cd /opt/cisco-aci-agent
$ ./launch configure --mode reset-settings --conf-path <directory path for settings.env/multi_site_config.json files (optional)>

Restart Collector

To restart the collector, execute the following commands. Also, the changes made in the settings.env and multi_site_config.json files are reflected in the collector after the restart.

$ cd /opt/cisco-aci-agent
$ ./launch restart-collector

Remove Unused Objects

To remove unused SDN objects, execute the following command.

$ launch delete-objects [--conf-path <configuration directory>]
Additional options and/or flags can be obtained from CLI help
$ launch delete-objects --help

Handle ASCII Code Encoding Issue

Readline library that Python uses to read interactive input does not accept non-ASCII characters sent by the terminal. It happens due to any of the following.

  • Terminal emulator does not provide characters in the UTF-8 encoding process.
  • Readline is not configured to accept UTF-8 input.

To overcome this issue, execute the following command before importing OOTB reports.

$ export LC_ALL='en_US.utf8'