MustGather: Information to collect when having HCX (Hybrid Cloud Extension) issues

Answer

Answer

For all HCX Issues:

1- Gather the Support Information from the source side. (screenshot needed for build, system ID and link last communicated info)

From the source vSphere Web Client -> HCX plug-in -> Support -> Support Information

2- Confirm Site Pairing is up

From the source vSphere Web Client -> HCX plug-in -> Dashboard -> Site Pairings


Note: If Site Pairing is failing, it could be a Proxy or firewall blocking traffic.

Confirm that port 443 from the HCX Manager on the source side to the site pairing URL is open by going to the CLI of the HCX Manager and telnet to the remote side URL on port 443.
- If this test fails, open firewall ports or configure Proxy to allow 443 traffic.
- If the error message indicates an untrusted connection you will need to add the remote HCX Manager’s certificate:
From the source HCX Manager (https://hcx-manager:9443) > Administration > Certificate > Trusted CA Certificate > URL > “remote side hcx manager URL”

3- Are HCX Tunnels Up/Green? (screenshot)

From the source vSphere Web Client -> HCX plug-in -> Interconnect -> HCX Components

4- Are all services running on both source and target sides? (screenshots)

From the HCX Appliance Management UI (https://hcx-manager:9443) -> Appliance Summary
SNMP and SSH are optional, all others should be in Running State 


5- Provide Uptime, CPU and memory usage for both HCX Managers (source and target) - screenshots

From the HCX Appliance Management UI (https://hcx-manager:9443) -> Dashboard

6- Is the connectivity to NSX and vCenter showing green/connected?

From the HCX Appliance Management UI (https://hcx-manager:9443) -> Dashboard

7- Is DNS name resolution working on both source and target?

1. SSH into the HCX Manager (or console) using the admin account
2. Switch User to root
3. Ping the VC and SSO URL/PSC by name.

8- Is time synchronized between the source and target HCX Managers?

1. SSH into the source HCX Manager (or console) using the admin account
2. Switch User to root
3. Issue the date command
4. SSH into the target HCX Manager (or console) using the admin account
5. Switch User to root
6. Issue the date command
7. Compare the date/time

9- On both source and target HCX Manager, review the logs.

Log location on HCX Manager: /common/logs/admin
Logs: app.log, web.log

Are there any errors worth noting?

Keywords to search for: FAIL in caps, exception, ERROR, migration

10- Gather HCX Manager Logs on both sides.

This can be done from the HCX Appliance Management UI (https://hcx-manager:9443) or the HCX plugin if available

From the HCX Appliance Management UI (https://hcx-manager:9443) -> Administration -> Troubleshooting -> Technical Support Logs


From the vSphere Web Client -> HCX plug-in -> Administration -> Troubleshooting

Select the logs to generate/request, if the problem is with the HCX appliances, include logs for them too.
Click on Generate, then download the logs.

These may be quite large so you will need to share them via your preferred repository so we can download them from there for review.



CGW, L2C, WAN-OPT issues:

If the issues are with HCX Interconnect services or Appliances: HCX Interconnect, HCX WAN-OPT, HCX Network Extension (L2C)


In addition to the above 10 items, also provide:

1- Appliance connection status from both source and target side:

1. SSH into the HCX Manager
2. Switch User to root
3. Type: ccli
4. Type: list
5. Are all HCX appliances showing Connected?

If not, verify TCP-443 is open between HCX Manager and the listed HCX Interconnect VMs.

2- If tunnels are down, confirm required ports are opened up between source and target:



Assuming ports are opened and ssh service is started, connect to HCX Manager via ssh as admin, then su to root:
Run these commands:
1. ccli
2. list
3. go <HCX appliance ID>
4. debug remoteaccess enable
5. ssh

Check the connectivity status from the HCX IC:
1. Check the Security associations are up: ipsec status
2. confirm the routing is appropriate: ip route
3. get the remote side IP: ip tunnel
4. try pinging the remote side (only if ICMP is not disallowed): ping <remote side IP>
5. confirm traffic on UDP port 500 and 4500 is sent and received: tcpdump -s0 -n -i vNic_# udp

(note the vNic_# might be vNic_0 or another vNic, you can get the info via "ip route" and use that in the tcpdump command above)