IBM Support

IBM AIX: tcpdump command fails with "bpf_load: genminor failed" error

How To


Summary

The tcpdump command fails with one of:
tcpdump: bpf_load: genminor failed: No such file or directory
tcpdump: bpf_load: genminor failed: Error 0
tcpdump: no suitable device found

Steps

The tcpdump command can be one of:
tcpdump -i <interface ex:en0> host <IP address> -w <file name>
tcpdump
The error usually indicates tcpdump cannot find any bpf (Berkeley Packet Filter) devices to run against or mismatch of the device major number in the ODM and the /dev directory.
Before checking for bpf devices, make sure network interfaces are online. For active connections, the netstat command should show contents of various network-related data structures. 
# netstat -in
The hardcoded limit for max number of bpf devices is 20.
Before making any changes to the ODM, backup the ODM CuDvDr.
# cp /etc/objrepos/CuDvDr /tmp/
# odmget CuDvDr > /tmp/odmget-CuDvDr.out
Step 1:
If there are any bpf minor devices in the ODM for the bpf major number, with "value3" set to something other than "bpf", remove them both from the ODM and /dev directory. For instance, for bpf major number 42, the non-bpf minor device instance is "uuvvlv_xyz6".
 $ cat odmget-CuDvDr.out | grep -ip bpf
CuDvDr:
        resource = "ddins"
        value1 = "bpf"
        value2 = "42"
        value3 = ""
CuDvDr:
        resource = "devno"
        value1 = "42"
        value2 = "1"
        value3 = "uuvvlv_xyz6"
 
Remove each one of these non-bpf minor devices in ODM:
# odmdelete -q value3=<ex: uuvvlv_xyz6> -o CuDvDr   
Then, remove the associated character and block non-bpf minor files for the bpf major number from the /dev directory.
Ex:
5th column below is the bpf major number.
# ls -l /dev | awk '/42/{print $0}'
crw-rw----    1 root     system       42,  2 Jun 22 17:46 ruuvvlv_xyz6
brw-rw----    1 root     system       42,  1 Jun 22 17:46 uuvvlv_xyz6
# rm /dev/ruuvvlv*
# rm /dev/uuvvlv*

Next, check for any existing bpf files in "/dev" with non-bpf device major number. If any exist without the associated major number device object in the ODM, remove them.

For instance, for bpf device in ODM with major number 42:
# odmget CuDvDr | grep -ip "value1\ =\ \"bpf"
CuDvDr:
        resource = "ddins"
        value1 = "bpf"
        value2 = "42"
        value3 = ""

if any bpf files exist in "/dev" directory with non-bpf major number, for ex:51:
# ls -al /dev/bpf*
example:
cr--------    1 root     system       51,  0 Jul 26 22:28 /dev/bpf0
cr--------    1 root     system       51,  1 Jul 26 22:28 /dev/bpf1

if there are no associated major number devices in ODM:
# odmget CuDvDr | grep -ip "value2\ =\ \"51"
<blank output>
Delete all the bpf files in the "/dev" directory for that major number
# rm /dev/bpf<n>        replace "n" with the bpf minor device number  ex: bpf0, bpf1 and so on.
Once the ODM and /dev directory are cleared of non-bpf minor devices for bpf major number and bpf files with non-bpf major number with no associated devices in the ODM, save the changes and reboot the LPAR
# savebase
# reboot
and then run:
# tcpdump -D
If problem persists, skip to data collection in Step 3.

Step 2:
Either if the LPAR cannot be rebooted now or if the scenarios in the step 1 do not exist, go through these instructions.  If problem persists, LPAR must be rebooted.
a)  For the bpf major number in CuDvDr, if some or none of the bpf minor devices and files exist in the ODM and /dev, run :
# tcpdump -D
If the bpf minor devices still do not get generated in the ODM, as a workaround, create bpf files in /dev directory and verify if the issue gets resolved. If not, reboot the LPAR
# mknod /dev/bpf<n>   c  <bpf major number ex: 42>  <n>    
# mknod /dev/bpf<n+1> c  <bpf major number>         <n+1>
# mknod /dev/bpf<n+2> c  <bpf major number>         <n+2>
# mknod /dev/bpf<n+3> c  <bpf major number>         <n+3>
Then, set permissions
# chmod 400 /dev/bpf<n>
# chmod 400 /dev/bpf<n+1>
# chmod 400 /dev/bpf<n+2>
# chmod 400 /dev/bpf<n+3>
Where n is the bpf minor device number.  If there are no bpf minor devices in CuDvDr, n can be zero, otherwise, n is the highest bpf minor device number that exists plus one. For instance, for bpf major device "36" in ODM that already has existing minor devices bpf0 through bpf6, next minor bpf device that can be created is bpf7, so, n is 7.
bpf device major number:
# odmget CuDvDr | grep -ip "value1\ =\ \"bpf" | grep -i value2
  value2 = "36"
existing bpf minor device numbers:
# odmget CuDvDr | grep -ip "value1\ =\ \"<bpf major number from output above, ex:36>" | grep -i value2
value2 = "0"
....
value2 = "6"

b) However, if the max number of 20 bpf minor devices already exist for the bpf major number, try to free up one of the bpf minor devices being used.
First, check for and stop the Iptrace if already running:
# lssrc -a | grep -i iptrace
# stopsrc -s iptrace
If IPtrace is started from the command line, kill it:
# ps -ef | grep -i iptrace
# kill -9 <iptrace process pid>
Unload the kernel extension loaded by the iptrace daemon regardless of IPtrace is running or not:
# iptrace -u
Run the command multiple times until following message is output:
iptrace: unload success!
If IPtrace is not running at all and problem persists even after kernel extension unload, try to free up  one of the bpf devices being used.
List out the process PIDs that are using the bpf minor devices
# fuser -cux /dev/bpf*
Verify the process PIDs and stop one of the process:
# ps -ef | grep <Process PID from the above output>
# kill -9 <Process PID>

If problem persists, reboot the LPAR.
After reboot, if the issue persists even after running one of these commands, skip to data collection in Step #3:
# tcpdump
or
# /usr/lib/methods/cfgbpf   

Step 3:
-- Remove existing snap
# snap -r  
                  
-- Create directory for data collection
# mkdir -p /tmp/ibmsupt/testcase
-- Start script session
# script /tmp/ibmsupt/testcase/'hostname'-script.out
-- Enable syslog debug logging
# cp /etc/syslog.conf /etc/syslog.conf.bak
# echo "*.debug /tmp/ibmsupt/testcase/syslog.out" >> /etc/syslog.conf
# touch /tmp/ibmsupt/testcase/syslog.out
Make the syslogd read the changes to syslog.conf
# refresh -s syslogd
Get bpf devices and files in ODM and /dev directory
# odmget CuDvDr > /tmp/ibmsupt/testcase/before-CuDvDr.out
# ls -l /dev/bpf*
Run tcpdump command:
# tcpdump -D
Test the tcpdump command that failed:
# <run the tcpdump command that failed>
Check again for bpf devices in ODM
# ls -l /dev/bpf*
# odmget CuDvDr > /tmp/ibmsupt/testcase/after-CuDvDr.out

Collect snap data
# snap -aZc
Exit script session
# exit
Rename the generated snap file
# mv /tmp/ibmsupt/snap.pax.Z /tmp/ibmsupt/<Case number>-snap.pax.Z
Upload  /tmp/ibmsupt/<Case number>-snap.pax.Z file.

Additional Information

SUPPORT:

If additional assistance is required after completing all of the instructions provided in this document, please follow the step-by-step instructions below to contact IBM to open a case for software under warranty or with an active and valid support contract.  The technical support specialist assigned to your case will confirm that you have completed these steps.

a.  Document and/or take screen shots of all symptoms, errors, and/or messages that might have occurred

b.  Capture any logs or data relevant to the situation.

c.  Contact IBM to open a case:

   -For electronic support, please visit the IBM Support Community:
     https://www.ibm.com/mysupport
   -If you require telephone support, please visit the web page:
      https://www.ibm.com/planetwide/

d.  Provide a good description of your issue and reference this technote

e.  Upload all of the details and data to your case

   -You can attach files to your case in the IBM Support Community
   -Or Upload data to IBM testcase server analysis:

    http://www.ibm.com/support/docview.wss?uid=ibm10733581

f.  Click here to submit feedback for this document.

[{"Type":"MASTER","Line of Business":{"code":"LOB08","label":"Cognitive Systems"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8mKe000000CaZBIA0","label":"Communication Applications-\u003EBPF\/IPTRACE"}],"ARM Case Number":"TS009045878","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions"}]

Document Information

Modified date:
11 December 2024

UID

ibm16841759

Page Feedback