SD-WAN VeloCloud Troubleshooting Guide

Edit online

This document provides some useful troubleshooting scenarios for SD-WAN VeloCloud Collector.

Note: Troubleshooting scenarios are listed in alphabetical order.

Troubleshooting

Edit online

Contains the following topics.

System and Kubernetes Log Collection Scripts
Device or Orchestrator populated with Partial Object synchronization
Upgrade process from version 6.8 / 7.0.0 / 7.1.0 / 7.2.0 to version 8.0.0
VeloCloud Pod Panic Error
How to change log level?

System and Kubernetes Log Collection Scripts

Edit online

Collecting journalctl logs ensures that system-level information is available. In scenarios where no pods are running during a failure, these logs provide critical insights for troubleshooting.

For system logs / journal logs,

Collect logs for the k3s service.
```
sudo journalctl -u k3s -f
```
Collect logs from the k3s service since the last system boot.
```
journalctl -u k3s --since "boot"
```

For Kubernetes logs,

Get details of the Kubernetes service.
```
kubectl get svc kubernetes -o wide
```

Get endpoints of the Kubernetes service.

kubectl get endpoints kubernetes -o wide

Get all nodes with labels.
```
kubectl get nodes --show-labels -o wide
```
Checks all nodes.
```
kubectl get pods -A
```

Device or Orchestrator populated with Partial Object synchronization

Edit online

If the Device Onboarding process is not functioning, check the spk import status in the MySQL database.

Navigate to SevOne NMS container.
```
nms
```
Execute the following MySQL config command to perform database validation.
```
mysqlconfig
```

Execute the SQL command as shown below.


SELECT vendor, spk_file_name, CAST(processed_status AS INT) AS processed_status,
CAST(active AS INT) AS active_int, processed_at
FROM net.spk_data
WHERE vendor = 'Velocloud';

Verify the import status for the .spk files.

Example


+-----------+------------------------------------------+------------------+------------------+------------+---------------------+
| vendor    | spk_file_name                            | processed_status | active_int | processed_at        |
+-----------+------------------------------------------+------------------+------------------+------------+---------------------+
| Velocloud | VeloCloudSDWAN_Alert_Policies_v1.0.0.spk |                1 |          1 | 2025-05-14 03:20:10 |
| Velocloud | VeloCloudSDWAN_Certification_v1.0.0.spk  |                1 |          1 | 2025-05-14 03:20:10 |
| Velocloud | VeloCloudSDWAN_MIBs_v1.0.0.spk           |                1 |          1 | 2025-05-14 03:20:08 |
| Velocloud | VeloCloudSDWAN_ObjectGroups_v1.0.0.spk   |                1 |          1 | 2025-05-14 03:20:10 |
| Velocloud | VeloCloudSDWAN_TopNViews_v1.0.0.spk      |                1 |          1 | 2025-05-14 03:20:20 |
+---------+--------------------------------------------+------------------+------------+------------+---------------------+
5 rows in set (0.001 sec)

Note: The processed_status should be set to 1 , and the processed_at field should contain the UTC timestamp when the device was first discovered.

Note: For Velocloud_* .spk files where processed_at is 0 or null, manually import them using the following command.

cd /config/sdwan/Velocloud

SevOne-import --allow-overwrite --file <spk_name>

Upgrade process from version 6.8 / 7.0.0 / 7.1.0 / 7.2.0 to version 8.0.0

Edit online

If QoE data fails to generate after the upgrade, perform the following steps.

Check VeloCloud logs.

/var/log/SDWAN/VeloCloud1/8.0.0-build.<###>/QoeAgent/VeloCloud1_QoeAgent_8.0.0-build.<###>.log

Check for the error mentioned below.

2024-05-28T13:23:33Z ERR SOA request error in /sevone.api.v3.Devices/Data
error="rpc error: code = Unauthenticated desc = failed to authenticate"

2024-05-28T13:23:33Z ERR Error in objectTransport.CreateDeviceData
error="rpc error: code = Unauthenticated desc = failed to authenticate"

2024-05-28T13:23:33Z INF Sending SOA request...
agent=QoeAgent
endpoint=/sevone.api.v3.Metadata/ObjectTypes
requestId=1

Restart SOA container.
```
podman restart nms-nms-soa
```
Retrieve VeloCloud API key from Redis.
- To retrieve VeloCloud API key.
```
podman exec -it nms-nms-redis redis-cli keys "velocloudapiKeyV3*"
 
```
- Access Redis Container.
```
podman exec -it nms-nms-redis redis-cli
 
```
- Delete VeloCloud API key from Redis (refer to step Retrieve VeloCloud API key from Redis above).
```
DEL <VeloCloud key retrieved from previous step>
 
```

VeloCloud Pod Panic Error

Edit online

If QoE data fails to generate, then execute the following steps.

Check VeloCloud pod logs.


podman logs nms-collections-velocloud-collector

Check if VeloCloud pod is restarting with the error below.


{"level":"info","ts":1717052925.4193544,"msg":"Data in redis needs to be updated"}
panic: runtime error: index out of range [0] with length 0goroutine 1 [running]:

If error is found then,

Retrieve VeloCloud-config key from Redis.


podman exec -it nms-nms-redis redis-cli keys "VeloCloud1-config*"

Access Redis container.


podman exec -it nms-nms-redis redis-cli

Delete the VeloCloud API key retrieved. Refer to step 3 (a).
```
DEL <VeloCloud key retrieved from previous step>
 
```

How to change log level?

Edit online

By default, the collector log level is set to info. For troubleshooting purposes, you may need to increase the log level to capture additional debug information. In such cases, follow the steps provided to change the log level to debug.

SSH into the SevOne NMS Host.
```
ssh <nms-user>@<nms-host-ip>
```
Enter the SevOne NMS Application Container
```
podman exec -it nms-nms-nms bash
```

Edit the collector configuration file using the editor of your choice.

vi /config/collectors/<collector-type>/<collector-name>/default-<collector-name>.yaml

Example:

vi /config/collectors/sdwan/VeloCloud/default-VeloCloud.yaml

Locate the log: section and change the log level from info to debug, as shown below.
```
 
log:
  level: debug
 
```
Note:
The collector supports the following log levels:
- info
- debug
After collecting logs, revert debug back to info.
Save the configuration. The changes are applied during the next discovery cycle.
Note: Running the collector at the debug log level is not recommended for extended periods. After you collect the required debug logs, or as directed by IBM SevOne Support, revert the log level to the default setting by using the same procedure.