IBM Support

NSX edge blocking or improperly filtering traffic

Question & Answer


Question

When using VMWare Solutions on IBM Cloud, you find that an NSX edge that is part of the route for traffic is blocked or dropped. For example, one of the following symptoms might occur:
  • The NSX edge is the default route for a virtual machine (VM) and is expected to be enabling outbound access from that VM to a target IP.  But, pinging that target IP from the VM fails.
  • The NSX edge is part of the network route between a physical Fortigate firewall and the private network. The private network addresses cannot be pinged from the Fortigate firewall.
  • An IPSec VPN tunnel using an NSX edge gateway with a local perimeter firewall has been established. The tunnel is up from both sides; however, you cannot ping a target in the IBM Cloud from the source side of the tunnel.
What options are there for problem determination and debugging for these types of cases?

Answer

To help isolate the issue, it is a good idea to debug directly from the NSX Edge (ESG).  First, enable SSH and connect to the ESG. This procedure does not require a redeploy and can be done without impact.
Steps to enable SSH:
  • In vCenter WebUI, log in as administrator@vsphere.local
  • Go to Networking & Security -> NSX Edges
  • Double-click the Edge associated with this issue
  • Go to tab: Manage->Settings->Configuration
  • Click the Actions pull-down menu at the top of the screen
  • Use the "Change CLI Credentials" action (click the Actions pull-down menu at the top of the screen to enable SSH and set the password)
Once you SSH-ed onto the edge to look at the traffic, you can issue commands such as follows:
Display all traffic through vNic_0 except SSH session traffic:
  • debug packet display interface vNic_0 port_not_22
To see all the traffic on vNic_0 for a certain source or destination IP address:
  • debug packet display interface vNic_0 net_<IP_address>
To see all the traffic on vNic_0 from 10.123.45.67:
  •  debug packet display interface vNic_0 net_10.123.45.67
To see all the traffic on vNic_1 from 10.123.45.67:
  •  debug packet display interface vNic_1 net_10.123.45.67

Other useful NSX edge commands:
  • (Details in NSX Command Line Interface Reference)
  • show ip route
  • show interface
    (Note: Use this command to see which interfaces to trace: vNic_0, vNic_1, etc)
  • ping
  • traceroute
  • show configuration static-routing
  • show configuration nat
  • show configuration interface
The VMware documentation, Edge Routing Connectivity Issues, has more details to help with troubleshooting edge routing issues.
If tracing on the NSX edge indicates traffic is not even reaching the NSX edge, tracing on the ESXi host where a virtual machine that is the source of traffic might be helpful. For example, for a VM that cannot successfully ping 8.8.8.8, try these commands:
On the VM:
  • ping 8.8.8.8
Use the following commands on the host where the VM is running:
Find the switchport for the VM:
  • net-stats -l
Capture traffic leaving the VM on that switchport:
  • pktcap-uw --switchport <switchport-from-net-stats-output> --dir 0 --stage 0 -o - | tcpdump-uw -enr - icmp
This pktcap-uw command might well yield plenty of output showing traffic.   Note this command specifies "--stage 0", which means the output should show traffic before firewall rules are applied.
   
Capture traffic leaving the switchport after the firewall had done its filtering:
  • pktcap-uw --switchport <switchport-from-net-stats-output> --dir 0 --stage 1 -o - | tcpdump-uw -enr - icmp
If this pktcap-uw command yields almost no output, that indicates that no traffic is leaving after the firewall applied filters.  Note that this command specifies "--stage 1" which means the output should show traffic after firewall rules are applied.
   
Output in the first step (stage 0) and almost no output in step 2 (stage 1) can indicate that the NSX Firewall might have a rule blocking traffic.   Review the firewall rules.

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSCLB3","label":"VMware Solutions"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"","label":""}}]

Document Information

Modified date:
07 April 2020

UID

ibm16152529

Page Feedback