IBM Support

The system is sending excessive IPv6 DNS queries.

Troubleshooting


Problem

Network or DNS admins are reporting that an AIX system is sending excessive IPv6 DNS queries.

Resolving The Problem


1- Make sure that the name resolution order is set to suppress IPv6 lookups:

# grep -v ^# /etc/netsvc.conf
hosts = local, bind4

2- Unset NSORDER system variable if it's already set
# echo $NSORDER
local,bind
# export NSORDER=""
# echo $NSORDER
#

3- Make sure that netcd daemon is not running and configured for a resolver.
# lssrc -s netcd

When the netcd daemon is running and configured for a resolver (for example, DNS) and a map (for example, hosts), the resolution is first made using the cached answers. If it fails, the resolver is called and the response is cached by the netcd daemon.

Check to see if you have/etc/netcd.conf file configured.
# grep -v ^# /etc/netcd.conf
#

By default, everything in that file is commented out.

If the file is not configured the netcd daemon should be stopped.
# stopsrc -s netcd

Comment out the start of netcd if it is uncommented.
# vi /etc/rc.tcpip
#start /usr/sbin/netcd "$src_running"


4- Some applications may not honour the netsvc.conf /etc/netsvc.conf name resolution order and use their own. To know which application is sending those DNS queries you would need to follow this procedure:

- Start an iptrace which looks only at port 53 for DNS
# startsrc -s iptrace -a "-a -p 53 -b /tmp/iptrace.trc"

- Start a kernel trace
# trace -a -j 101,104,106,107,10C,134,139,252,255,465,536,539 -T 80000000 -L 160000000 -o /tmp/ktrace.bin ; sleep 10 ; trcstop

- Stop the iptrace
# stopsrc -s iptrace

- Collect process table output
# ps -ef > /tmp/ps-ef.out

- Format the kernel trace
# trcrpt -O pid=on,exec=on /tmp/ktrace.bin > /tmp/ktrace.out

- Use Wireshark or ipreport to find the IPv6 dns requests (type=AAAA) note the local port number and convert it to hex
i.e port=44360 --> 0xAD48

Example of how to convert the above port to HEX
# bc -l
obase=16 <enter>
44360 <enter>
AD48
<ctrl D> to exit

*NOTE: Wireshark can be downloaded online and is free. If you decide to go with ipreport, then use this command to convert from binary to text format:
# ipreport -rns /tmp/iptrace.bin > /tmp/iptrace.txt

- Search in the kernel trace using this local port number (lport) number obtained above (e.g. AD48)
# vi /tmp/iptrace.txt

539 dcm 2289732 51912861 0.000860537 0.001744 PCB return from in_pcbhashlookup2 _faddr=F1000200067B981A fport=0000000000000035_laddr=F1000200067B981E lport=000000000000AD48

there you can see the process and parent ID's.

- Now you can search the ps -ef output (saved earlier) for these process ID's to get the corresponding process name and thus find the problem application.
# vi /tmp/ps-ef.out

[{"Product":{"code":"SWG10","label":"AIX"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Not Applicable","Platform":[{"code":"PF002","label":"AIX"}],"Version":"Version Independent","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
17 June 2018

UID

isg3T1026923