How to start kernel trace during boot?

(1) Run ctctrl command to set trace level of appropriate component to systracemax (i.e. 9). The component name will be provided by the AIX support team.

(2)

(a) Create a special directory to save the trace files. e.g. /tmp/ibm
Note: Make sure that /tmp/ibm has plenty of free space.

(b) Add "trace" command as shown below red in the /sbin/rc.boot immediately before the line
"cfgmgr -p2 -v 2>&1 | alog -t boot # key is in normal position".
Note: The hooks will be provided by the AIX support team.

=== start of /sbin/rc.boot ===
...
...
trace -C all -a -l n -j (add trace hooks provided by the AIX support team here) -L 40000000 -T 20000000 -o /tmp/ibm/trc.bin

cfgmgr -p2 -v 2>&1 | alog -t boot # key is in normal position
...
...
=== end of /sbin/rc.boot ===

(3) Add "trcstop" command at the very bottom of /etc/rc.tcpip file.

(4) Reboot the system.

(5) Multiple files will be created in /tmp/ibm directory. Send all files in /tmp/ibm directory to AIX Support along with full snap for analysis.

(6) Once debugging is done, set component trace level to normal.

Example 1:

Configuration:
- AIX system has 10 Gb ethernet adapter configured as ent0.
- ent0 PCIe2 4-Port Adapter (10GbE SFP+) (e4148a1614109304)
- Feature Code/Marketing ID: EN0S
- trace hooks: 105d,105e,105f,1061

This 10 Gb adapter uses driver "shientdd" and there is a need to trace events of this driver at boot time.

Steps:
(1) set component trace for shientdd to 9.
ctctrl systracemax -c shientdd -r

(2) Add "trace" command as shown below in red in the /sbin/rc.boot.

(a) mkdir /tmp/ibm
Note: Make sure that /tmp has plenty of free space

(b) trace -C all -a -l -n -j 105d,105e,105f,1061 -L 40000000 -T 20000000 -o /tmp/ibm/trace.bin

(3) Add "trcstop" command at the very bottom of /etc/rc.tcpip file.

(4) Reboot the system.

(5) Multiple file will be created in /tmp/ibm directory. Send all files in /tmp/ibm directory along with full snap to AIX Support for analysis.

(6) Once debugging is done, set component trace level to normal.
ctctrl systracenormal -c shientdd -r

Example 2:

Configuration:
- AIX system has etherchannel device configured.
- ent4 EtherChannel / IEEE 802.3ad Link Aggregation
- trace hooks: 2FA,2FB

This etherchannel device uses driver "ethchandd" and there is a need to trace events of this driver at boot time.

Steps:
(1) set component trace for etherchannel to 9.
ctctrl systracemax -c etherchannel -r

(2) Add "trace" command as shown below in red in the /sbin/rc.boot.

(a) mkdir /tmp/ibm
Note: Make sure that /tmp has plenty of free space

(b) trace -C all -a -l -n -j 2FA, 2FB -L 40000000 -T 20000000 -o /tmp/ibm/trace.bin

(3) Add "trcstop" command at the very bottom of /etc/rc.tcpip file.

(4) Reboot the system.

(5) Multiple file will be created in /tmp/ibm directory. Send all files in /tmp/ibm directory along with full snap to AIX Support for analysis.

(6) Once debugging is done, set component trace level to normal.
ctctrl systracenormal -c etherchannel -r