SD-WAN VeloCloud Solution Deployment / Configuration Guide

About

This document describes the steps to deploy and configure the VeloCloud solution.

Prerequisites

  • An administrator-level account in SevOne NMS.
  • SSH password for the support account.
  • IP address of the PAS.

Installation Steps

SevOne NMS

The following steps apply to perform an installation from scratch of the VeloCloud solution on SevOne NMS.

Important:

If you are on VeloCloud version 7.0.2 or above, skip Steps 2, 3, 4, and 7 below.

  1. Using ssh, login to SevOne NMS appliance as support.
    $ ssh support@<SevOne NMS appliance IP address>
  2. Change directory to /config/collectors/sdwan/velocloud/on the NMS appliance host.
    $ cd /config/collectors/sdwan/velocloud/
  3. Download the following (latest) files from IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact IBM SevOne Support for the latest files.You must place <tar/zip> files in /config/collectors/sdwan/velocloud directory.
    1. sdwan-velocloud-installation-v7.0.0-build.<###>.tgz
    2. sdwan-velocloud-installation-v7.0.0-build.<###>.tgz.sha256.txt
    3. signature-tools-<latest-version>-build.<latest>.tgz
    4. signature-tools-<latest-version>-build.<latest>.tgz.sha256.txt
  4. Execute the following commands to verify the checksum of the code signing tool before extracting it.
    $ (cd /config/collectors/sdwan/velocloud && cat $(ls -Art signature-tools-*.tgz.sha256.txt | \
    tail -n 1) | sha256sum --check)
    
    $ sudo tar xvfz $(ls -Art /config/collectors/sdwan/velocloud/signature-tools-*.tgz | \
    tail -n 1) -C /
  5. Go to SevOne NMS container.
    $ sudo podman exec -it <nms_container_id_or_name> /bin/bash
  6. Change directory /config/collectors/sdwan/velocloud/on the NMS pod.
    $ cd /config/collectors/sdwan/velocloud/
  7. Verify the signature of Solutions .tgz. files.
    $ sh /usr/local/scripts/SevOne-validate-image -i $(ls -Art /config/collectors/sdwan/velocloud/sdwan-*.tgz | \
    tail -n 1) -s $(ls -Art /config/collectors/sdwan/velocloud/sdwan-*.tgz.sha256.txt | tail -n 1)
  8. Extract the latest build.
    $ tar -xvf $(ls -Art /config/collectors/sdwan/velocloud/sdwan-*.tgz | tail -n 1) -C /config/collectors/sdwan/velocloud

    You will see the following files in the folder.

    • VeloCloudSDWAN.Certification.spk - it creates one device type VeloCloudSDWAN and 5 object types suffixed with (VeloCloud).
  9. Replace the file VeloCloudSDWAN.Certification.spk in the spk directory /config/collectors/sdwan/velocloud/spk.
    $ cp VeloCloudSDWAN.Certification.spk /config/collectors/sdwan/velocloud/spk
  10. Change directory to /config/collectors/sdwan/velocloud/spk.
    $ cd /config/collectors/sdwan/velocloud/spk

    You will see the following files in the directory.

    • VeloCloudSDWAN.MIBs.spk - it imports two VeloCloudSDWAN MIB files (VELOCLOUD-EDGE-MIB.mib and VELOCLOUD-MIB.mib).
    • VeloCloudSDWAN.Certification.spk - it creates one device type VeloCloud SDWAN and 5 object types suffixed with (VeloCloud).
    • VeloCloudSDWAN.DeviceGroups.spk - it creates 1 device group.
    • VeloCloudSDWAN.ObjectGroups.spk - it creates 3 object group class (VeloCloud SDWAN) and 3 Object Groups underneath it.
    • VeloCloudSDWAN.MetadataSchema.spk - it imports the metadata schema for VeloCloud SDWAN devices.
    • VeloCloudSDWAN.Alert.Policies.spk - it imports 5 alert policies. All policies are imported as disabled by default.
    • VeloCloudSDWAN.TopNViews.spk - it imports 22 TopN Report views.
    • VeloCloudSDWAN-DI-OOTB-Reports.tar - it imports 1 SevOne Data Insight report and 3 Templates.
  11. Import the following .spk files in sequence.
    1. VeloCloudSDWAN MIBs
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.MIBs.spk
    2. Device Type and Object Types
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.Certification.spk
    3. Metadata Schema
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.MetadataSchema.spk
    4. Device Groups
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.DeviceGroups.spk
    5. Object Groups
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.ObjectGroups.spk
    6. Alert Policies
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.Alert.Policies.spk

      The following is the list of alerts imported.

      • VeloCloud SDWAN - WAN Link - Packet Loss
      • VeloCloud SDWAN - WAN Link - Latency
      • VeloCloud SDWAN - WAN Link - Jitter
      • VeloCloud SDWAN - VCG Path - Latency
      • VeloCloud SDWAN - VCG Path - Packet Loss
      Important: All alerts are disabled by default.

Device Onboarding

To onboard VeloCloud devices in SevOne NMS, follow these steps.

  1. Enter the URL for the SevOne NMS appliance into your web browser to display the Login page. Enter your credentials on the login page and click Login.VeloCloud NMS Login Page VeloCloud NMS Home Page
  2. From the navigation bar, click the Devices menu and select Device Manager. VeloCloud Device Manager
  3. Click Add Device to display the New Device page. VeloCloud Add Device
  4. On the New Device page, please add the following details. VeloCloud New Device
    1. In the Name field, enter the device name.
    2. In the Alternate Name field, enter an alternate device name. You can search for a device by its alternate name.
    3. In the Description field, enter the device description. You can use this to provide additional information about the function, location, or any other pertinent information about the device.
    4. In the IP Address field, enter the device IP address.
    5. Click the plugin drop-down. By default, it is set to SNMP. Select SDWAN.
      1. Select the Enable SDWAN API Integration check box.
        VeloCloud SDWAN Plugin
      2. Click the Vendor drop-down and select the VeloCloud option.
      3. In the VCO URL field, enter the URL for SDWAN vendor, VeloCloud.
      4. In the Username field, enter the username for SDWAN vendor, VeloCloud.
      5. In the Password field, enter the password for SDWAN vendor, VeloCloud.
      6. In the Poll Frequency field, enter how often to poll the device. The default value is 10 minutes.
      7. Enable field Auto-discover and monitor associated VeloCloud Edges - Use SNMP Plugin to automatically discover and monitor VeloCloud devices.
    6. Click Save As New to save the current changes as a New Device.
    7. Once the SDWAN plugin is configured, from the plugin drop-down, select plugin SNMP. VeloCloud SNMP Plugin
    8. Ensure that the field SNMP Capable check box is selected to enable the discovery of SNMP object types and to poll SNMP data on the device.
    9. Enter credentials (Username & Password) for VeloCloud devices. (Make sure to have same SNMP credentials for all VeloCloud Devices).
    10. Select other options and click Save As New to save the current changes as a New Device. This device is then queued for discovery.
    11. A new device has been added to the Device Manager screen.
    12. Click the Devices menu and select Discovery Manager. Here, you will see the device is in the discovery queue.
    13. After the discovery process is completed, VeloCloud devices will be visible on the Device Manager screen. VeloCloud Devices
    14. To retrieve the metadata of a VeloCloud SDWAN device, follow these steps.
      1. From the list, select a device to view its metadata.
      2. Click Edit metadata in the Actions column to open the Edit Metadata pop-up.
      3. In the Edit Metadata pop-up, locate the section VeloCloud SDWAN DEVICE to find the metadata fields. VeloCloud Metadata
        Important: Value for the following metadata attributes might differ in VeloCloud and SevOne NMS.
        • Authentication
        • Service Up Since
        • Software Updated
        • System Up Since
Note: TopN Report Views - Import on SevOne NMS
$ SevOne-import --allow-overwrite --file VeloCloudSDWAN.TopNViews.spk

The following is the list of TopN reports imported.

  • VeloCloud SDWAN CPU Utilization
  • VeloCloud SDWAN HA Peer State
  • VeloCloud SDWAN Link Jitter
  • VeloCloud SDWAN Link Latency
  • VeloCloud SDWAN Link Packet Loss
  • VeloCloud SDWAN Link Packet Loss Pct
  • VeloCloud SDWAN Link State
  • VeloCloud SDWAN Link Utilization
  • VeloCloud SDWAN Links
  • VeloCloud SDWAN Memory Utilization
  • VeloCloud SDWAN Path State
  • VeloCloud SDWAN Path Utilization
  • VeloCloud SDWAN Paths Jitter
  • VeloCloud SDWAN Paths Latency
  • VeloCloud SDWAN Paths Packet Loss
  • VeloCloud SDWAN QoE After Ehancements (Video)
  • VeloCloud SDWAN QoE After Enhancements (Voice)
  • VeloCloud SDWAN QoE After Enhancements (Transactional)
  • VeloCloud SDWAN Link QoE Before Enhancements (Voice)
  • VeloCloud SDWAN Link QoE Before Enhancements (Video)
  • VeloCloud SDWAN Link QoE Before Enhancements (Transactional)
  • Device ICMP Reachability

SevOne Data Insight - OOTB Reports

There are two methods to import the OOTB reports.
  1. Import via CLI
  2. Import via SevOne Data Insight
Once the OOTB reports are imported via one of the two methods mentioned above, the following list of reports are imported.
  • Indicator Summary Calendar View
  • VeloCloud SDWAN Path Summary
  • VeloCloud SDWAN Link Summary
  • VeloCloud SDWAN Site Summary
  • VeloCloud SD-WAN Dashboard
  • Indicator Summary (Percent %)
  • Indicator Summary (Bytes)
  • Indicator Summary
  • Object Summary
  • Device Summary

Import via CLI

  1. Using ssh, login to SevOne NMS appliance as support.
    $ ssh support@<SevOne NMS appliance IP address>
  2. Go to SevOne NMS container.
    $ sudo podman exec -it <nms_container_id_or_name> /bin/bash
  3. Change directory to /config/collectors/sdwan/velocloud/spk.
    $ cd /config/collectors/sdwan/velocloud/spk
  4. Copy VeloCloud-DI-OOTB-Reports.tar to your SevOne Data Insight machine.
    $ scp VeloCloudSDWAN-DI-OOTB-Reports.tar sevone@<DI IP Address>:<path to directory>

    Example

    $ scp VeloCloudSDWAN-DI-OOTB-Reports.tar sevone@10.49.14.8:/home/sevone/
  5. Load reports.
    $ sevone-cli sdi reports load <REPORTS-TAR-FILE-PATH>

Import via SevOne Data Insight

  1. Download VeloCloudSDWAN-DI-OOTB-Reports.tar file from your SevOne NMS appliance and save it locally.
  2. Log in to your SevOne Data Insight machine by navigating to the appropriate URL in your browser.
  3. On the Report Manager screen, click Import button. VeloCloud SD-WAN Import Button
  4. Click or drag file to upload. For example, VeloCloudSDWAN-DI-OOTB-Reports.tar.
    1. Select an apt datasource from the Datasource drop-down.
    2. Select the Assign each report to its original owner's username check box to assign the reports imported to its original owner's username.
    3. Click Upload.
      VeloCloud SD-WAN Import OOTB
      Note:
      • Reports can only be imported from a .tar file. Other file extensions are not acceptable. If the file extension is not a .tar file then it will simply ignore the action.
      • Reports can be imported to the same or newer version of SevOne Data Insight as the one they were exported from, by drag and drop into Reports.
      • SevOne does not support the importing of reports from a newer to older version.

Upgrade

To upgrade your VeloCloud solution on SevOne NMS, follow the steps below.

Important:

If you are on VeloCloud version 7.0.2 or above, skip Steps 3, 4, 5, and 8 below.

  1. If your SevOne NMS is not already on version 7.0, proceed to upgrade it to 7.0.
  2. Using ssh, login to SevOne NMS appliance as support.
    $ ssh support@<SevOne NMS appliance IP address>
  3. Change directory to /config/collectors/sdwan/velocloud/on the NMS appliance host.
    $ cd /config/collectors/sdwan/velocloud/
  4. Download the following (latest) files from IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact IBM SevOne Support for the latest files.You must place <tar/zip> files in /config/collectors/sdwan/velocloud directory.
    1. sdwan-velocloud-installation-v7.0.0-build.<###>.tgz
    2. sdwan-velocloud-installation-v7.0.0-build.<###>.tgz.sha256.txt
    3. signature-tools-<latest-version>-build.<latest>.tgz
    4. signature-tools-<latest-version>-build.<latest>.tgz.sha256.txt
  5. Execute the following commands to verify the checksum of the code signing tool before extracting it.
    $ (cd /config/collectors/sdwan/velocloud && cat $(ls -Art signature-tools-*.tgz.sha256.txt | \
    tail -n 1) | sha256sum --check)
    
    $ sudo tar xvfz $(ls -Art /config/collectors/sdwan/velocloud/signature-tools-*.tgz | \
    tail -n 1) -C /
  6. Go to SevOne NMS container.
    $ sudo podman exec -it <nms_container_id_or_name> /bin/bash
  7. Change directory to /config/collectors/sdwan/velocloud/on the NMS pod.
    $ cd /config/collectors/sdwan/velocloud/
  8. Verify the signature of Solutions .tgz files.
    $ sh /usr/local/scripts/SevOne-validate-image -i $(ls -Art /config/collectors/sdwan/velocloud/sdwan-*.tgz | \
    tail -n 1) -s $(ls -Art /config/collectors/sdwan/velocloud/sdwan-*.tgz.sha256.txt | tail -n 1)
  9. Extract the latest build.
    $ tar -xvf $(ls -Art /config/collectors/sdwan/velocloud/sdwan-*.tgz | tail -n 1) -C /config/collectors/sdwan/velocloud

    You will see the following files in the folder.

    • VeloCloudSDWAN.Certification.spk - it creates one device type VeloCloudSDWAN and 5 object types suffixed with (VeloCloud).
  10. Replace the file VeloCloudSDWAN.Certification.spk in the spk directory /config/collectors/sdwan/velocloud/spk.
    $ cp VeloCloudSDWAN.Certification.spk /config/collectors/sdwan/velocloud/spk
  11. Change directory to /config/collectors/sdwan/velocloud/spk.
    $ cd /config/collectors/sdwan/velocloud/spk

    You will see the following files in the directory.

    • VeloCloudSDWAN.MIBs.spk - it imports two VeloCloudSDWAN MIB files (VELOCLOUD-EDGE-MIB.mib and VELOCLOUD-MIB.nib).
    • VeloCloudSDWAN.Certification.spk - it creates one device type VeloCloud SDWAN and 5 object types suffixed with (VeloCloud).
    • VeloCloudSDWAN.DeviceGroups.spk - it creates 1 device group.
    • VeloCloudSDWAN.ObjectGroups.spk - it creates 3 object group class (VeloCloud SDWAN) and 3 Object Groups underneath it.
    • VeloCloudSDWAN.MetadataSchema.spk - it imports the metadata schema for VeloCloud SDWAN devices.
    • VeloCloudSDWAN.Alert.Policies.spk - it imports 5 alert policies. All policies are imported as disabled by default.
    • VeloCloudSDWAN.TopNViews.spk - it imports 22 TopN Report views.
    • VeloCloudSDWAN-DI-OOTB-Reports.tar - it imports 1 SevOne Data Insight report and 3 Templates.
  12. Import the above .spk files in sequence.
    1. VeloCloudSDWAN MIBs
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.MIBs.spk
    2. Device Type and Object Types
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.Certification.spk
    3. Metadata Schema
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.MetadataSchema.spk
    4. Device Groups
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.DeviceGroups.spk
    5. Object Groups
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.ObjectGroups.spk
    6. Alert Policies
      $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.Alert.Policies.spk

      The following is the list of alerts imported.

      • VeloCloud SDWAN - WAN Link - Packet Loss
      • VeloCloud SDWAN - WAN Link - Latency
      • VeloCloud SDWAN - WAN Link - Jitter
      • VeloCloud SDWAN - VCG Path - Latency
      • VeloCloud SDWAN - VCG Path - Packet Loss
      Important: All alerts are disabled by default.
  13. Perform the steps provided in the section Device Onboarding if you have not done so previously.
    Important:
    • Devices that were added manually in the past are not currently linked to the orchestrator. Any devices that were absent previously but are now present in the orchestrator will be automatically discovered and linked to the orchestrator during the discovery process.
    • After the VCO device discovery process is completed, it populates the value for metadata attribute Orchestrator name for VeloCloud devices that were previously onboarded. Run the device discovery process for all existing devices to populate values for all metadata attributes associated with those devices. For the newly added devices by VCO, the metadata population happens once the discovery process is completed for them.
  14. Import the TopN reports on SevOne NMS container using CLI.
    
    $ SevOne-import --allow-overwrite --file VeloCloudSDWAN.TopNViews.spk
    

    The following is the list of TopN reports imported.

    • VeloCloud SDWAN CPU Utilization
    • VeloCloud SDWAN HA Peer State
    • VeloCloud SDWAN Link Jitter
    • VeloCloud SDWAN Link Latency
    • VeloCloud SDWAN Link Packet Loss
    • VeloCloud SDWAN Link Packet Loss Pct
    • VeloCloud SDWAN Link State
    • VeloCloud SDWAN Link Utilization
    • VeloCloud SDWAN Links
    • VeloCloud SDWAN Memory Utilization
    • VeloCloud SDWAN Path State
    • VeloCloud SDWAN Path Utilization
    • VeloCloud SDWAN Paths Jitter
    • VeloCloud SDWAN Paths Latency
    • VeloCloud SDWAN Paths Packet Loss
    • VeloCloud SDWAN QoE After Ehancements (Video)
    • VeloCloud SDWAN QoE After Enhancements (Voice)
    • VeloCloud SDWAN QoE After Enhancements (Transactional)
    • VeloCloud SDWAN Link QoE Before Enhancements (Voice)
    • VeloCloud SDWAN Link QoE Before Enhancements (Video)
    • VeloCloud SDWAN Link QoE Before Enhancements (Transactional)
    • Device ICMP Reachability

Solution Verification and Customization

Perform the following steps to log onto your SevOne NMS appliance. For more details, please refer to SevOne NMS System Administration Guide or SevOne NMS User Guide > section Login.

  1. Enter the URL for the SevOne NMS appliance into your web browser to display the Login page.
  2. Enter the credentials and click Login. For example, Username: admin and Password: SevOne
  3. To check MIB files imported, click the Administration menu, select Monitoring Configuration, and then select MIB Manager. For more details on MIB Manager, please refer to SevOne NMS System Administration Guide > section MIB Manager. VeloCloud SD-WAN MIB Manager
  4. To check device groups imported, click the Devices menu and select Grouping, then Device Groups. For more details on Device Groups, SevOne NMS User Guide > section Device Groups. VeloCloud SD-WAN Device Groups
  5. To check object groups imported, click the Devices menu, select Grouping , and then select Object Groups. For more details on Object Groups, SevOne NMS System Administration Guide > section Object Groups. VeloCloud SD-WAN Object Groups
    Important: You can change the Object Group Membership Rules based on your network environment.
  6. Check VeloCloud SDWAN OOTB reports imported on SevOne Data machine. VeloCloud SD-WAN OOTB Reports

    The following is the list of reports imported.

    • Indicator Summary Calendar View
    • VeloCloud SDWAN Path Summary
    • VeloCloud SDWAN Link Summary
    • VeloCloud SDWAN Site Summary
    • VeloCloud SD-WAN Dashboard
    • Indicator Summary (Percent %)
    • Indicator Summary (Bytes)
    • Indicator Summary
    • Object Summary
    • Device Summary

Troubleshooting

Upgrade process from version 6.7 and version 6.8 to version 7.0.0

If QoE data fails to generate post upgrade, then execute the steps below.

  1. Check VeloCloud logs.
    $ /var/log/VeloCloud1/7.0.0-build.6/QoeAgent/VeloCloud1_QoeAgent_7.0.0-build.6.log
  2. Check for the error mentioned below.
    {"name":"Orchestrator (VCO)","type":"Generic","old_ts":1716901953,"new_ts":1716901953,"ip":"0.0.0.0","automatic_creation":true,"objec2024-05-28T13!https://emoji.slack-edge.com/T27SFGS2W/12/9885a4ebae28a36a.png!33Z ERR SOA request error in /sevone.api.v3.Devices/Data error="rpc error: code = Unauthenticated desc = failed to a2024-05-28T13!https://emoji.slack-edge.com/T27SFGS2W/12/9885a4ebae28a36a.png!33Z ERR Error in objectTransport.CreateDeviceData error="rpc error: code = Unauthenticated desc = failed to authenti2024-05-28T13:23:33Z INF Sending SOA request... agent=QoeAgent endpoint=/sevone.api.v3.Metadata/ObjectTypes requestId=1
  3. Restart SOA container.
    $ podman restart nms-nms-soa
  4. RetrieveVeloCloud API key from Redis.
    • To Retrieve VeloCloud API Key.
      $ podman exec -it nms-nms-redis redis-cli keys "velocloudapiKeyV3*"
    • Access Redis Container.
      $ podman exec -it nms-nms-redis redis-cli
    • Delete VeloCloud API key from Redis (refer to step :To retrieve VeloCloud API key from Redis).
      $ DEL<VeloCloud key retrieved from previous step>

VeloCloud Pod Panic Error

If QoE data fails to generate, then execute the following steps.
  1. Check VeloCloud pod logs.
    $ podman logs nms-collections-velocloud-collector
  2. Check if VeloCloud pod is restarting with below error.
    {"level":"info","ts":1717052925.4193544,"msg":"Data in redis needs to be updated"}
    panic: runtime error: index out of range [0] with length 0goroutine 1 [running]:
    
  3. If error is found,
    • Retrieve VeloCloud-config key from Redis.
      $ podman exec -it nms-nms-redis redis-cli keys "VeloCloud1-config*"
    • Access Redis container.
      $ podman exec -it nms-nms-redis redis-cli
    • Delete the VeloCloud API key retrieved (refer to step :To retrieve VeloCloud API key from Redis).
      $ DEL<VeloCloud key retrieved from previous step>