Troubleshooting
Problem
Environment
Resolving The Problem
The steps below will walk you through on how to set up a web log monitor as well as a HTTP Server trace leveraging the trace watch functionality.
Identify the error message or string to monitor for in your HTTP Server or Application Server log file
You can monitor for any message or string that is specific to the issue occurring with your HTTP Server or Application Server instance. Once message or string is logged in the file being monitored, an HTP8989 message will be thrown to the QSYSOPR message log. This message or string should be unique to the problem you are experiencing.
Typically, you will be monitoring for a message thrown to either of these log files:
- /www/<server>/logs/error_log.Qxxxxxxxxx
- /www/<server>/wlp/usr/servers/<server>/logs/messages.log
- /QIBM/UserData/WebSphere/AppServer/<version>/<edition>/profiles/<profileName>/logs/<serverName>/SystemOut.log file.
Thread exception example:
[mpm_worker:notice] [pid 79:tid 0000002F] ZSRV_MSG0372: QHTTPSVR/QZSRHTTP: Thread Exception occurred: : t.
For the above message, we could select the string "ZSRV_MSG0372" or "Thread Exception occurred" to monitor for. I always recommend the more unique string. So, I will monitor for "ZSRV_MSG0372" in this scenario and example.
Set up the Web Log Monitor for your HTTP Server or Application Server instance
NOTE: If monitoring the HTTP Server's access or error log files, you might need to reconfigure the Web Log Monitor every morning since the access and error logs are typically cycled to a new file every 24 hours.
a) Open the IBM Web Administration for i console. (http://<server>:2001/HTTPAdmin or https://<server>:2010/HTTPAdmin)
b) Click on Manage -> HTTP Servers or Application Servers.
c) Select the HTTP Server or Application Server you would like to monitor.
d) Click on "Web Log Monitor" link under Tools on the bottom, left-hand corner of the page.
e) Click on the "Enable Web Log Monitor" button.
f) Browse for the log file in the IBM i IFS you would like to monitor. (i.e. /www/myHTTPServer/logs/error_log.Q119081500)
g) Click the Next button.
h) On the Basic tab, configure the string/error message you would like to monitor for.
"ZSRV_MSG0372"
The Advanced tab provides additional filter options.
i) Click the Next button.
j) Review the Web Log Monitor configuration summary. Click the Finish button to complete the set-up.
The screen will refresh and display the configured web log monitor information.
Configure the Web Log Monitor to send a notification message to QSYSOPR
a) In the IBM i Web Administration console, go to Manage -> HTTP Servers or Application Servers -> Web Log Monitor (under Tools).
b) Click on the Notification tab when managing your Web Log Monitor.
c) Check the box next to "Send message to *QSYSOPR message queue", if this is not already checked.
You can also click the "Send Test Notification" button to simulate the HTP8989 message to QSYSOPR message queue. After clicking the button, execute the CL command "DSPMSG QSYSOPR" to view the QSYSOPR message log.
Message ID . . . . . . : HTP8989 Severity . . . . . . . : 40
Message type . . . . . : Information
Date sent . . . . . . : 08/15/19 Time sent . . . . . . : 09:36:32
Message . . . . : Web Log Monitor test message.
Cause . . . . . : This message is used as a general diagnostic message
d) Click the Apply button to commit the changes.
Configure the Web Log Monitor Interval
a) In the IBM i Web Administration console, go to Manage -> HTTP Servers or Application Servers -> Web Log Monitor.
b) Click on the Miscellaneous tab when managing your Web Log Monitor.
c) Under "Monitor interval", set the value for "Check the log file every" to the desired value in minutes.
We recommend anywhere from a 1 -5 minute interval depending on how frequently the issue is occurring and how quickly we need to stop the trace to catch the issue. Please note that the lower the check interval, the more CPU resources required to monitor the log file.
e) Click the Apply button to commit the changes.
Start the HTTP Server Trace Watching for the HTP8989 Web Log Monitor Message
TRCTCPAPP APP(*HTTP) SET(*ON) TRCFULL(*WRAP) HTTPSVR(<instance>) TRCLVL(*VERBOSE) WCHMSG((HTP8989)) WCHMSGQ((*SYSOPR))
Where <instance> is the name of the IBM HTTP Server instance you configured the Web Log Monitor against.
After starting the HTTP Server trace with the watch functionality, the TRCTCPAPP utility will automatically end once the HTP8989 message is thrown to the QSYSOPR message log. The message is thrown once the Web Log Monitor detects the configured string/expression in the log file being monitored. In my example, the TRCTCPAPP trace will end once the string "ZSRV_MSG0372" string is detected in the /www/APACHEDFT/logs/error_log.Q119081500 file.
The trace watch functionality is also available for several other IBM i OS trace utilities including TRCCNN, TRCINT, STRCMNTRC, and STRTRC as well as other application types with the TRCTCPAPP utility.
Verify the Trace Watch Exists or Has Completed
WRKWCH WCH(*TRCCMD)
You should see a watch entry with an Origin of TRCTCPAPP (or whatever trace utility you are using) for the User that executed the trace command. The status of the watch should be Active. If you don't see a watch entry, the watch was likely triggered or ended for some reason (like an IPL). IBM recommends ending all traces prior to an IPL (i.e. TRCTCPAPP APP(*HTTP) SET(*END)). The Web Log Monitor can also be disabled, but should survive an IPL. After the IPL has completed, everything can be restarted.
The TRCTCPAPP will produce one or more QZSRHTTPTRC spool files under the IBM i user profile that started the TRCTCPAPP trace utility. Other trace utilities will create different output data in various formats. Please refer to the documentation for each specific trace utility regarding the output data produced.
Related Information
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
18 December 2019
UID
ibm10964912