Question & Answer
Question
You have an instance of IBM Cloud for VMware Solutions vCenter Server and have been using vCenter for weeks. After you add a new firewall to the environment, however, you cannot access vCenter. How can you debug this problem?
Answer
The firewall might be interfering with network traffic between the ESXi servers and the NFS storage for the management virtual machines (VMs) such as the vCenter VM.
To check, log on to the ESXi hosts. (Credentials are provided in the IBM Cloud for VMware Solutions portal on the Infrastructure tab.)
Issue the following command:
- esxcli storage nfs list
The command output includes columns that shows if the storage is accessible and mounted and the IP address of the NFS server, for example:
[root@host1:~] esxcli storage nfs list
Volume Name Host Share Accessible Mounted Read-Only
---------------- ------------ ------------ ---------- ------- ---------
workload_share_1 10.xxx.xx.xx /IBM0..._3/ true true false
workload_share_0 10.xxx.xx.xx /IBM0..._2/ true true false
management-share 10.xxx.xx.xx /IBM0..._1/ true true false
Volume Name Host Share Accessible Mounted Read-Only
---------------- ------------ ------------ ---------- ------- ---------
workload_share_1 10.xxx.xx.xx /IBM0..._3/ true true false
workload_share_0 10.xxx.xx.xx /IBM0..._2/ true true false
management-share 10.xxx.xx.xx /IBM0..._1/ true true false
The output from the following command can help verify the IP addresses used to access storage:
- esxcli network ip interface ipv4 address list
A simple test for connectivity to the NFS server (10.xx.xx.xxx) on port 2049 (the standard NFS port):
- nc -z 10.xx.xx.xx 2049
Testing with ping and traceroute to the NFS server address using the specific interface for NFS traffic:
- ping 10.xx.xx.xx -I vmk2
- traceroute 10.xx.xx.xx -i vmk2
Other symptoms with NFS storage connectivity can include All Paths Down (APD) messages in the VMware logs, such as /var/log/vmkernel.log. Sample messages:
- APD start event for 0xxxxxxxxx [xxx]
- Device or filesystem with identifier [xxx] has entered the All Paths Down state.
If the NFS storage connectivity looks correct, review other possible causes for vCenter connection problems in Not able to connect to vCenter.
You can also check that the ESXi hosts are authorized to access the NFS storage. In IBM Cloud, select Classic Infrastructure > Storage > File Storage. Then click the storage LUN to see the authorized subnets.
Access from the ESXi host to NFS storage is accomplished on a private VLAN separate from the private VLAN used for VMware managment traffic. To see how the NFS traffic is routed, use the following steps:
- esxcli storage nfs list
- Look for the host name of the NFS server in the storage list, for example, fsf-xxxx.adn.networklayer.com
- Ping the NFS server and make note of the IP address, for example, 161.aa.bbb.cc
- esxcli network ip route ipv4 list
- Note that there is a route with Interface: vmk3 and Network: 161.aa.bbb.0 and a private gateway address 10.ddd.eee.ff
The vmk3 Interface route specifies the subnet for NFS server IP address in the Network column, 161.aa.bbb.0, and ensures that NFS traffic uses the vmk3 vmkernel interface. The default route uses the vmk0 vmkernel interface and is on a subnet on a different VLAN, used for management traffic. Management and storage traffic are on different subnets on different VLANs.
The Gateway, 10.ddd.eee.ff, on the vmk3 Interface route matches the subnet used by vmk3. To confirm this setup, use the following steps:
- esxcli network ip interface ipv4 address list
- Note that in the list of vmkernel IP addresses, vmk3 has an IP address, 10.ddd.eee.ff, in the same subnet as the route Gateway noted earlier
- This subnet, 10.ddd.eee.ff, is the subnet that is authorized to access the NFS storage
- Access to the NFS server through the authorized subnet should work:
- vmkping -I vmk3 fsf-xxxx.adn.networklayer.com
- Access to the NFS server through the any other subnet (for example, vmk0) should fail:
- vmkping -I vmk0 fsf-xxxx.adn.networklayer.com
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSCLB3","label":"VMware Solutions"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"","label":""}}]
Was this topic helpful?
Document Information
Modified date:
07 April 2020
UID
ibm16172557