Sitworld: The Encyclopedia of ITM Tracing and Trace Related Controls
John Alvord, IBM Corporation
Draft #2 – 23 September 2016 - Level 1.01000
ITM tracing is at once the most tiresome of topics and sometimes the most important. This post will collect everything I have collected and discovered over the years. Expect to see many additions and corrections over time.
Chapter 1 is dedicated to controlling diagnostic log file sizes.
Chapter 2 is dedicated to describing operation log files - what they are and where they are found.
Further chapters are being prepared. There is also some testing to be performed and some of the model control files might change.
Chapter I - Control of diagnostic log file size and location.
ITM Diagnostic log file size and location control differ by platform [Linux/Unix, Windows, z/OS and i/5]. This diagnostic log information contains detailed process information. By default – when control set to ERROR – you see error messages and any information level messages. The error messages do not always mean an actual error condition. The entire goal is to help IBM Support to understand product problem issues. This chapter shows how to control the size and location.
All of the examples assume the default install location. In practical usage you will specify the directories actually chosen for the particular installation.
Following is the best practice from ITM 623 GA onward. Earlier best practice is described at the end.
- In the /opt/IBM/ITM/config directory create a file xx.environment which has the same attributes/owner/group as xx.ini. For example, lz.environment and lz.ini or ms.environment and ms.ini. If one already exists, just use it. Here are example commands to create a new such environment file.
- cd /opt/IBM/ITM/config
- ls -l ms.ini which showed -rw-rw-rw- 1 itmuser staff 2565 Aug 5 21:17 ms.ini
- touch ms.environment
- chmod 666 ms.environment
- chown itmuser ms.environment
- chgrp staff ms.environment
- These environment variables are applied just before starting the process else and will take effect without any reconfiguration. A recycle is needed of course. When testing is over they can be commented out or deleted or the whole file deleted.
- Other environment variables can be specified, but we are working here on just KBB_RAS1_LOG where characteristics like diagnostic log segment sizes are defined. If this is absent, ITM has some built in defaults.
- Following are the environment variable lines to enable KBB_PRAS1_LOG. Further explanations follow. The lines are separate by blank lines here for clarify but blank lines are not required.
RUNNINGHOSTNAME=`hostname|cut -d. -f1`
KBB_RAS1_LOG=%(CTIRA_LOG_PATH)/%(RUNNINGHOSTNAME)_%(PRODUCTCODE)_%(syspgm)_%(sysutcstart)-.log INVENTORY=%(CTIRA_LOG_PATH)/%(RUNNINGHOSTNAME)_%(PRODUCTCODE)_%(syspgm).inv COUNT=16 LIMIT=20 PRESERVE=1 MAXFILES=64
- CTIRA_LOG_PATH needs to be set to directory where the logs will be kept. One large customer uses /var/opt/IBM/ITM/logs
- KBB_VARPREFIX needs to be set to allow ITM basic services to recognize substitutions. If % is already in use, [such as in LDAP environment variables] you can use ! or %%.
- RUNNINGHOSTNAME This will calculate the hostname the ssame way normal configuration does. It avoids then need for a reconfigure in the process
- .PRODUCTCODE this should be set to the product code, like lz for Linux or ms for TEMS.
- Following are the elements of the KBB_RAS1_LOG. Some are the environment variables specified about and are not commented on.
- COUNT number of diagnostic log segments, 16 maximum. I consider 3 a logical minimum.
- LIMIT size in megabytes of diagnostic log segments. I once had to use 200 megs and 16 could to capture 24 hours log.
- PRESERVE=1 make sure the first segment is preserved
- MAXFILES total number of files to be kept. This *must* be larger than COUNT
- %(syspgm) will be replaced by main program like klzagent or kdsmain
- %(sysutcstart) will be replaced with start epoch UTC time in seconds
- The maximum space used will be MAXFILES*LIMIT in megabytes.
Before ITM 623 GA, best practice was as follows:
- Create a file xx.override in /opt/IBM/ITM/config with the same attributes/owner/group as xx.ini.
- Put within that file the change you want to introduce. Compared to modern setups the values need to be specified within single quotes such as
- Instead of calculating RUNNINGHOSTNAME and PRODUCT on the fly, instead figure out what they should be and add them to the KBB_RAS1_LOG definition.
- In the xx.ini file add the following line
This is known as a source include definition.
- For many agents this is sufficient. TEMS is a instanced agent and more work is needed.
- The first method is to do a ./itmcmd config -S -t <temsname> and accept all defaults.
- The second method is to add the source include line into the hostname_ms_temsname.config file at the end.
Windows diagnostic log segments have the same form as Linux/Unix.
Tracing is defined via the Managed Tivoli Enterprise Monitoring GUI.
- Right click on agent [such as TEMS or Windows OS Agent]
- Select Advanced
- Select Edit Trace Parms…
There are entry areas for
Maximum Log Size Per File [LIMIT]
Maximum Number of Log Files Per Session [COUNT]
Maximum Number of Log Files Total [MAXFILES]
As usual a recycle is needed to implement.
The diagnostics log information is included in the RKLVLOG SYSOUT file.
When intensive tracing is configured this can grow to a large size. If that happens, the following command will close the current SYSOUT file and start another. In that way the current log can be captured to a disk file. That is often performed using SDSF after the switch to “print” to a disk file.
/f cmstask,TLVLOG SWITCH
The CLASS option can be used to specify the output class of the new sysout file.
/f cmstask,TLVLOG SWITCH CLASS=W
z/OS logs are not configurable with KBB_RAS1_LOG. Configuration variables are kept in the RKANDATU PDS member KxxENV.
The i/5 platform [previously AS/400] uses the following form for KBB_RAS1_LOG. This environment variable is placed in QAUTOTMP/KMSPARM file member KBBENV.
(QAUTOTMP/KA4AGENT01 QAUTOTMP/KA4AGENT02 QAUTOTMP/KA4AGENT03)
KBB_RAS1_LOG is specific as one lone line although it is presented here on separate lines for clarity.
The meaning of the controls is identical to Linux Unix.
Chapter 2 Operations log
The operations log is a high level view of ITM operations. It bears a close relationship to the ITM Universal Message Console. Universal Messages are written into a wrap around in storage table and are also written to the disk log for later analysis. The interesting benefit here is that you can write situations against the Universal Message Console. Each ITM process has a universal message console and if you want more details see this document:
Linux/Unix ITM Operations Log
The TEMS operations log is contained in the /opt/IBM/ITM/logs directory and has the name format
<hostname> names the server the TEMS is running on
<epoch> is a decimal number corresponding to the POSIX epoch - the number of seconds since 1 January 1970 not counting leap seconds at the time the log started. Diagnostic logs use a hex representation of epoch in the logs.
ITM Agents on Linux/Unix are contained in the same /opt/IBM/ITM/logs directory and has the name
<agent_name> is the managed system name [or Agent Name]
During each startup, the current <agent_name>.LG0 is copied to <agent_name>.LG1.
The Warehouse Proxy Agent operations log is not kept as a disk file. Those logs are preserved in a data warehouse table.
Windows ITM Operations Log
The Windows TEMS operations log is in the C:\IBM\ITM\cms directory and has the name kdsmain.msg.
The Windows Agent operations log has the same name format as Linux/Unix but any colons [:] in the agent name are converted to underlines. That is required because the Windows operating system does not permit colons in file names. The location depends on many factors including 32 versus 64 bit agent instance.
z/OS TEMS and Agent operations log
There is no separate log for z/OS operations log. Instead it is inter-mixed with diagnostic log lines in the RKLVLOG SYSOUT file.
i/5 Agent operations log
The operations log information is a message queue
Chapter 3 Diagnostic Trace Controls
[to be written]
Chapter 4 Implementing Diagnostic Trace Controls
[to be written]
Chapter 5 Communication tracing
[to be written]
Details on ITM tracing controls.
History and Earlier versions
Chapter 2 on Operations Log added
Correct Linux/Unix KBB_RAS1_LOG example