SD-WAN Viptela Collector Troubleshooting Guide

The following are some scenarios on how you may debug an issue while deploying SD-WAN Viptela Collector.

$ cat /SevOne.info
$ kubectl version o --short
$ helm version

$ docker ps | grep soa

$ kubectl get nodes
$ kubectl get pods
$ kubectl get pods -n=kube-system

$ cd /var/log/sdwan-viptela/<collector_name>/<build_version>
$ cat InstallerAgent/viptela_InstallerAgent_6.8.0-build.44.log

$ kubectl logs statefulset.apps/solutions-sdwan-viptela-redis-master --tail 100

# Using a text editor of your choice, you can find the logs in /var/log/soa.log
 
$ vi /var/log/soa.log

$ supervisorctl start jaeger
 
<Trigger the event to happen in SD-WAN Collector>
 
$ /usr/local/bin/jaeger

1. After creating the virtual machine, you may want to change its name. Execute the following steps.

   $ ssh sevone@<SD-WAN Viptela collector node IP address>
   
   $ sudo hostnamectl set-hostname "<enter hostname>"
   
   $ sudo reboot

2. When provisioning of control plane node is complete via user interface, ensure that control plane node is correctly provisioned from CLI.

   $ ssh sevone@<SD-WAN Viptela 'control plane' node IP address>
   
   $ kubectl get nodes
   
   NAME           STATUS   ROLES                 AGE    VERSION
   sdwan-node01   Ready    control-plane,master  2m45s  v1.27.1+k3s1

3. When the agent nodes have joined the Kubernetes cluster, execute the following command to confirm the same.

   $ ssh sevone@<SD-WAN Viptela collector 'control plane' node IP address>
   
   $ kubectl get nodes
   
   NAME           STATUS   ROLES                 AGE    VERSION
   sdwan-node01   Ready    control-plane,master  2m45s  v1.27.1+k3s1
   sdwan-node02   Ready    <none>                2m45s  v1.27.1+k3s1
   sdwan-node03   Ready    <none>                2m45s  v1.27.1+k3s1

4. To check the status of the deployment, ensure that all the pods are in Running status.

   $ kubectl get pods
   
   NAME                                                 READY   STATUS      RESTARTS   AGE
   solutions-sdwan-viptela-redis-node-0                 2/2     Running     0          8h
   solutions-sdwan-viptela-redis-node-1                 2/2     Running     0          8h
   solutions-sdwan-viptela-redis-node-2                 2/2     Running     0          8h
   solutions-sdwan-viptela-collector-58b8bb9c9-ts8m7    1/1     Running     0          8d
   solutions-sdwan-viptela-aug-7c4b59968c-vqp77         1/1     Running     0          8d
   solutions-sdwan-viptela-create-keys-1-zcrrz          0/1     Completed   0          8h
   

If a pod is ever stuck and you want it to reboot, you can append --grace-period=0 --force to the end of your delete pod command.

Example

$ ssh sevone@<SD-WAN Viptela collector 'control plane' node IP address or hostname>
  
$ kubectl delete pod $(kubectl get pods | grep 'dsm' | awk '{print $1}') --grace-period=0 --force

$ kubectl get pods -o wide

Output:

NAME                                                READY   STATUS             RESTARTS        AGE     IP               NODE        NOMINATED NODE   READINESS GATES
solutions-sdwan-viptela-aug-865df979c5-jshx5        1/1     Running            1 (6h21m ago)   6h24m   217.32.129.195   sevonek8s   <none>           <none>
solutions-sdwan-viptela-redis-node-1                2/2     Running            2 (6h21m ago)   6h24m   192.168.80.98    sevonek8s   <none>           <none>
solutions-sdwan-viptela-redis-node-0                2/2     Running            2 (6h21m ago)   6h24m   192.168.80.97    sevonek8s   <none>           <none>
solutions-sdwan-viptela-redis-node-2                2/2     Running            2 (6h21m ago)   6h24m   192.168.80.96    sevonek8s   <none>           <none>
solutions-sdwan-viptela-upgrade-2h7ml               0/1     Completed          0               4h49m   192.168.80.105   sevonek8s   <none>           <none>
solutions-sdwan-viptela-create-keys-10-n5gw8        0/1     Completed          0               4h49m   192.168.80.106   sevonek8s   <none>           <none>
solutions-sdwan-viptela-collector-98bc49c98-8t7zk   0/1     CrashLoopBackOff   10 (11s ago)    27m     192.168.80.108   sevonek8s   <none>           <none>

If the status of the collector pod is CrashLoopBackOff as shown above, it indicates the collector pod failure to connect to Redis pod.

error in agentRunState.Run when MustSucceed=true error="error in agent load: error in migrationManager.
Migrate: error in migrationManager.handler.Done: error in agentLastRun.ResetAll: error in cache.Del: error in redisClient.
Del: READONLY You can't write against a read only replica.

Execute the following commands on the master node of the collector to restart the Redis pod.

$ kubectl scale sts solutions-sdwan-viptela-redis-node --replicas=0

$ kubectl scale sts solutions-sdwan-viptela-redis-node --replicas=3

$ sevone-cli solutions reload

To ensure that the collector pods are running without any error, re-run the following command.

$ kubectl get pods -o wide

If you see the following error during an upgrade, run the command below to delete the cached metadata files.This will prompt yum to fetch new metadata the next time it runs.

TASK [Install convert2rhel and latest kernel] 
**************************************
**************************************
**************************************
fatal: [sevonek8s]: FAILED! => {"changed": false, "msg": 
"No package matching 'convert2rhel' found available, installed or updated", "rc": 126, "results": 
["All packages providing kernel are up to date", "No package matching 'convert2rhel' found available, installed or updated"]} 

$ sudo yum clean metadata 

If you are deploying the same build again or have updated /opt/SevOne/chartconfs/solutions-sdwan-viptela_custom_guii.yaml file, the following commands must be executed.

Applies only when configuration has been updated. The helm command uninstalls the deployment along with the base configuration which by default, is available with the .ova image file.

$ sevone-cli playbook precheck
 
$ sevone-cli solutions reload

To stop the collector, execute the following commands.

1. Using ssh, log into SD-WAN collector control plane node as sevone.

   $ ssh sevone@<SD-WAN Viptela collector 'control plane' node IP address or hostname>

2. Stop the collector.

   $ sevone-cli solutions stop_collector

If you are upgrading from SD-WAN 2.9 to SD-WAN version > 2.9, execute the following step. By default, the .ova image is already running a base configuration of the collector. If the configuration is modified, the following command must be executed.

$ sevone-cli solutions reload

Perform the following action if the agent nodes are in a Not Ready state after rebooting.

1. Using ssh , log into the designated SD-WAN collector control plane node and agent node as sevone from two different terminal windows.

   SSH to 'control plane' node from terminal window 1

   $ ssh sevone@10.123.45.67

   SSH to 'agent' node from terminal window 2

   $ ssh sevone@10.123.45.68

2. Obtain a list of DNS entries in /etc/resolv.conf file for both control plane and agent nodes in this example.

   From terminal window 1

   $ cat /etc/resolv.conf
   # Generated by NetworkManager
   search sevone.com nwk.sevone.com
   nameserver 10.168.16.50
   nameserver 10.205.8.50

   From terminal window 2

   $ cat /etc/resolv.conf
   # Generated by NetworkManager
   search sevone.com nwk.sevone.com
   nameserver 10.168.16.50
   nameserver 10.205.8.50

3. Ensure that DNS server can resolve SD-WAN collector's hostname / IP address on both the control plane and the agent nodes along with the DNS entries in /etc/resolv.conf file (see the search line and nameserver(s)).

   From terminal window 1

   The following output shows that the DNS server can resolve hostname / IP address on both the control plane and the agent nodes.

   Check if 'nslookup' resolves the 'control plane' IP address

   $ nslookup 10.123.45.67
   67.45.123.10.in-addr.arpa   name = sdwan-node01.sevone.com.

   Check if 'nslookup' resolves the 'control plane' hostname

   $ nslookup sdwan-node01.sevone.com
   Server:     10.168.16.50
   Address:    10.168.16.50#53
   
   Name:   sdwan-node01.sevone.com
   Address: 10.123.45.67

   Check if 'nslookup' resolves the 'agent' IP address

   $ nslookup 10.123.45.68
   68.45.123.10.in-addr.arpa   name = sdwan-node02.sevone.com.

   Check if 'nslookup' resolves the 'agent' hostname

   $ nslookup sdwan-node02.sevone.com
   Server:     10.168.16.50
   Address:    10.168.16.50#53
   
   Name:   sdwan-node02.sevone.com
   Address: 10.123.45.68

   nslookup name 'sevone.com' in search line in /etc/resolve.conf

   $ nslookup sevone.com
   Server:     10.168.16.50
   Address:    10.168.16.50#53
   
   Name:   sevone.com
   Address: 23.185.0.4

   nslookup name 'nwk.sevone.com' in search line in /etc/resolve.conf

   $ nslookup nwk.sevone.com
   Server:     10.168.16.50
   Address:    10.168.16.50#53
   
   Name:   nwk.sevone.com
   Address: 25.185.0.4

   nslookup nameserver '10.168.16.50' in /etc/resolve.conf

   $ nslookup 10.168.16.50
   50.16.168.10.in-addr.arpa   name = infoblox.nwk.sevone.com.

   nslookup nameserver '10.205.8.50' in /etc/resolve.conf

   $ nslookup 10.205.8.50
   50.8.205.10.in-addr.arpa    name = infoblox.colo2.sevone.com.

   From terminal window 2

   The following output shows that the DNS server can resolve hostname / IP address on both the control plane and the agent nodes.

   Check if 'nslookup' resolves the 'agent' IP address

   $ nslookup 10.123.45.68
   68.45.123.10.in-addr.arpa   name = sdwan-node02.sevone.com.

   Check if 'nslookup' resolves the 'agent' hostname

   $ nslookup sdwan-node02.sevone.com
   Server:     10.168.16.50
   Address:    10.168.16.50#53
   
   Name:   sdwan-node02.sevone.com
   Address: 10.123.45.68

   Check if 'nslookup' resolves the 'control plane' IP address

   $ nslookup 10.123.45.67
   67.45.123.10.in-addr.arpa   name = sdwan-node01.sevone.com.

   Check if 'nslookup' resolves the 'control plane' hostname

   $ nslookup sdwan-node01.sevone.com
   Server:     10.168.16.50
   Address:    10.168.16.50#53
   
   Name:   sdwan-node01.sevone.com
   Address: 10.123.45.67

   nslookup name 'sevone.com' in search line in /etc/resolve.conf

   $ nslookup sevone.com
   Server:     10.168.16.50
   Address:    10.168.16.50#53
   
   Name:   sevone.com
   Address: 23.185.0.4

   nslookup name 'nwk.sevone.com' in search line in /etc/resolve.conf

   $ nslookup nwk.sevone.com
   Server:     10.168.16.50
   Address:    10.168.16.50#53
   
   Name:   nwk.sevone.com
   Address: 25.185.0.4

   nslookup nameserver '10.168.16.50' in /etc/resolve.conf

   $ nslookup 10.168.16.50
   50.16.168.10.in-addr.arpa   name = infoblox.nwk.sevone.com.

   nslookup nameserver '10.205.8.50' in /etc/resolve.conf

   $ nslookup 10.205.8.50
   50.8.205.10.in-addr.arpa    name = infoblox.colo2.sevone.com.

   Important: If any of the nslookup commands in terminal window 1 or terminal window 2 above fail or return one or more of the following, you must first resolve the name resolution issue otherwise, SD-WAN collector will not work.

   Example

   ** server cant find 67.45.123.10.in-addr.arpa.: NXDOMAIN
   
   or
   
   ** server cant find 68.45.123.10.in-addr.arpa.: NXDOMAIN
   
   or
   
   *** Cant find nwk.sevone.com: No answer
   
   etc.

   If the name resolution fails due to any reason after the deployment of SD-WAN collector, then this could also lead to the failure of normal operations in SD-WAN collector. Hence, it is recommended to ensure that the DNS configuration is always working.

As a security measure, fresh installations do not ship with pre-generated SSH keys.

1. Using ssh, log into the SD-WAN Viptela collector control plane node as sevone.

   $ ssh sevone@<SD-WAN collector 'control plane' node IP address or hostname>

   Example

   $ ssh sevone@10.123.45.67

2. Execute the following command to generate unique SSH keys for your cluster.

   $ sevone-cli cluster setup-keys

To change the collector log level for any particular agent without redeploying the collector, perform the following steps.

1. Find redis master.

   $ kubectl exec -it solutions-sdwan-viptela-redis-node-0 --container sentinel -- redis-cli -p 26379
   
   $ SENTINEL get-master-addr-by-name redis-master

2. Get redis-cli shell on the master pod identified in Step-1.

   Example

   $ kubectl exec -it solutions-sdwan-viptela-redis-node-2 --container redis -- redis-cli

3. Publish debug message to the loggingCommand channel.

   PUBLISH loggingCommand "{agent-name}:{loglevel}:{logtype}:{minutes}"

   Where,

   • agent-name - Defines the name of the agent.
   • loglevel - Defines the log-level for the collector. Value can be info, debug, warning, or error.
   • logtype - Defines the type of logs. Value can be nms / vendor / all.
     • nms - Only NMS API response will be printed in debug logs.
     • vendor - Only vendor (vManage / vDirector) API logs will be printed in debug logs.
     • all - NMS and vendor logs will be printed in debug logs.
   • minutes - Defines time (in minutes) for which logs must be printed based on this message. For example, if minutes is set to 5, then debug logs will be printed for 5 minutes based on the message sent from the redis.

   Example

   PUBLISH loggingCommand "DeviceHealthStreamingAgent:debug:all:1"