Why does traceroute show different results for two AIX servers that have the same network configuration and AIX level?

Answer

The traceroute sends 3 udp probes to the destination host to 3 different ports 33435, 33436 and 33437 by default. 

If no application is listening on the port, the destination host responds with an ICMP “Destination Unreachable – Port Unreachable” message, and traceroute prints the round-trip time for that probe. Figure 4 shows udp probes and icmp port unreachable packets captured by tcpdump on Host B. The "netstat -Aan" output on Host B shows that no application is listening on UDP ports 33435, 33436, or 33437, so traceroute prints the round-trip time for all three probes, as shown in Figure 2.

On Host B
# tcpdump -n -i en0 host 10.0.0.30
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en1, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:12.608047 IP 10.0.0.30.33040 > 20.0.0.10.33435: UDP, length 1472
20:49:12.608101 IP 20.0.0.10. > 10.0.0.30: ICMP 20.0.0.10 udp port 33435 unreachable, length 36
20:49:12.608300 IP 10.0.0.30.33040 > 20.0.0.10..33436: UDP, length 1472
20:49:12.608324 IP 20.0.0.10. > 10.0.0.30: ICMP 20.0.0.10 udp port 33436 unreachable, length 36
20:49:12.608502 IP 10.0.0.30.33040 > 20.0.0.10..33437: UDP, length 1472
20:49:12.608526 IP 20.0.0.10 > 10.0.0.30: ICMP 20.0.0.10 udp port 33437 unreachable, length 36

                                                                                                          Figure 4

If the application is listening on the port, the destination host does not respond with an icmp "Destination Unreachable - Port Unreachable" message and the traceroute prints an * (asterisk) for that probe. The "netstat -Aan" output on Host B shows that the application is listening on UDP ports 33436 and 33437 as shown in Figure 5, so traceroute prints an asterisk (*) for two probes as shown in Figure 3.

On Host B:
# netstat -Aan | grep 3343
f100100007380600  udp4    0   0    *.33436     *.*
f100100007297600  udp4    0   0    *.33437     *.*

                                                  Figure 5

The lsof can be used to find which application is using udp port. The lsof is available at https://www.ibm.com/resources/mrs/assets/packageList?source=aixbp&lang=en_US. The lsof syntax is "lsof -n -P | grep <port number>". The application can be stopped to free up the port or run traceroute with -p to use different port range which is free.<port> to use different port range which is free. For example: "traceroute -p 40000 -n 20.0.0.15" uses ports 40001, 40002 and 40003 instead of 33435, 33436 and 33437.

Author: Darshan Patel
Platform: AIX on Power
Feedback: aix_feedback@wwpdl.vnet.ibm.com