Sitworld: TEMS Audit Process and Tool
Version 1.64000 26 May 2017
Follow on twitter
There have been cases every year where a TEMS was running with high enough CPU/Storage resource usage that the customer was concerned. In some cases, the TEMS experienced a steady storage growth and failure after some days. This condition is usually caused by workloads, configuration or environmental conditions. Situations are the most common workload but SOAP calls, historical data collection and Portal Client workspaces can be involved. In addition, there may be important error messages in the diagnostic log.
In 2008 I began work on a year-old customer problem that took until Spring 2010 to resolve. The conclusion was that a certain type of situation caused a severe storage fragmentation and a TEMS failure in 10-12 days. The customer decided to recycle the affected remote TEMSes once a week. That was probably the single most expensive PMR [for IBM and customer] I ever worked on.
Some 6 months later in Dec 2010 I had a customer with six AIX servers running at 95% utilization and all from remote TEMS processing. The solution was presented but the customer was unconvinced. I wrote a Perl program to summarize the results in a spreadsheet file. The customer was convinced, made the changes and those six systems dropped to 10% utilization. In March 2011, I published the process and tool as a technical note and it is now widely used.
Since the last published version 1.37000 [28 advisories] 38 new advisories have been added and many report sections to explain the environment further. At level 1.64000 the documentation has been reworked for simplicity and clarity.
- The TEMS Audit report itself includes an explanation of each advisory and that is not duplicated in this document. The temsaud.pl program can be viewed to see each advisory explanation.
- The separate word file documents have been eliminated and information added here, revised and extended.
- Each report section is documented along with needed traces.
These changes will continue. The source in every case is based on diagnosing a specific customer issue and when possible adds an advisory or report section which would speed the diagnostic effort – even allowing the customer to correct the condition whenever possible. In recent months there have been several advisories and/or report sections added each month.
TL;DR - Too Long; Didn't Read
Sometimes you want immediate results and to heck with the details. Here are the instant use instructions. Unpack the most recent zip file and copy the temsaud.pl and testaud.ini files to a convenient location where Perl is installed [all Unix and Linux environments] and on the system where a TEMS is running. Let’s pick /tmp and the TEMS is installed at the default directory. Now run this command
perl temsaud.pl -v -logpath /opt/IBM/ITM/logs
and if there should happen to be more than one candidate inventory file,
perl temsaud.pl -v -logpath /opt/IBM/ITM/log hostname_ms_kdsmain.inv
and then view the report file temsaud.csv. The end of that file includes an explanation of each advisory. If you want more information about the TEMS workload then add the needed traces documented later.
Linux and Unix systems usually have Perl installed.
On Windows, you can install the community edition at www.activestate.com. You can also just copy the needed files to a system where Perl is installed.
For a z/OS TEMS trace, capture the RKLVLOG sysout log into a file and run temsaud.pl with the -z option. I worked to make that automatic but have not found the secret sauce.
Perl is usually preinstalled in Linux/Unix systems. . For Windows you may need to install from www.activestate.com or any other source. Alternatively you can copy the needed log files to a system where Perl is installed. The program only uses Perl core services, No CPAN modules are needed.
TEMS Audit has been tested with
This is perl 5, version 20, subversion 1 (v5.20.1) built for MSWin32-x64-multi-thread
zLinux with Perl v5.8.7
There are two TEMS Audit program files in a zip file here
1) temsaud.pl Perl program
2) temsaud.ini Option file for setting nominal values
Install them somewhere convenient.
Run Time Options.
There are many but temsaud.pl does not require any of them.
-b Show HEARTBEATs in Managed System report
-cmdall When cmd trace is present, report all commands
-expslot <mins> Historical export report, display slot size - default 60 mins
-flip Produce detailed report on agent location flipping, ITM 630 FP7
-h display help information
-jitall produce advisory on minor level heartbeat jitter
-logpath Directory path to TEMS logs – default current directory
-o Specify the output report filename
-odir Specify directory where output files will be stored
-ossdir What directory to write send status report
-nohdr Remove variable header lines – used for regression testing
-rd Result detail reports
-ri <secs> result workload report by interval, default 60 seconds
-rdslot Seconds in result detail slots, default 60 seconds
-sqldetail produce a SQL detail report by source and table involved
-sr SOAP Report minute by minute - soap_detail.txt
-ss produce a report send status – used for multiple remote TEMSes
-sum produce a one line summary file
-v Produce some progress messages in STDERR
-work Copy files to work directory before analyzing.
Environment variable Windows – TEMP, Linux/Unix tmp
-workpath Directory path to work directory, default is the system
-z z/OS RKLVLOG log
The remaining parameter [if present] is a log file specification. Often no specification is needed. It will look at the -logpath setting and determine the current logs from the inventory file. Sometimes there is more than one candidate and you can specify it like
You can also specify log files to study as a partial name such as
That setting analyzes multiple log sections with the same file name starting string. That can be extremely useful of the .inv file is missing or was not sent with the diagnostic files to be studied. It is also the way to analyze a diagnostic log set that is not the most current.
In hands-free operation, do not specify a specific file. Instead use -logpath to specify a path to a normal TEMS logs directory or leave it off for the current directory. In hands-free mode the directory is examined to determine the current active log segments. If there are multiple .inv files the one with the most recent diagnostic logs is chosen. The log segments are processed in the sequential time order. When -work option is set, the diagnostic logs are copied to a work directory to avoid the issue where a log is deleted while being used. Specify the. inv file in case there are more than one candidate.
Note: If there is a one hour or higher time gap between the first and next segment, the first segment is processed only for certain messages. That said there are times when you will need to manually assemble a single combined log to analyze. That can be true if traces are turned on dynamically. Some of the reports use the time to calculate rates and isolating just the traced log makes that more accurate.
The default output report is temsaud.csv. Use the -o option to specify your own name for the report.
One section of the report is advisories. If the maximum advisory impact level is equal or greater than 50, the program ends with exit code 1 and otherwise 0. That can be used in shell files to control other actions – like sending email.
perl temsaud.pl cmsall.log
perl temsaud.pl -v
Reports and Diagnostic tracing needed
The reports are intended to be self-contained and easy to use. With no tracing beyond the default ERROR you get a lot of useful information. To get full value you need to set tracing. See post on how to set tracing. The following shows a report and how to get extra value by setting diagnostic tracing.
This labels the report and TEMS Audit version number, when run, the runtime parameters, the logs used [if -v is specified] and a line which is the TEMS nodeid and *LOCAL for hub TEMS or *REMOTE for remote TEMS. Occasionally the last line is missing because not in the logs. The header can be suppressed using the -nohdr option.
TEMS Audit report v1.63
Start: 2017-05-24 15:44:38
Runtime parameters: -sum -rd -flip -odir /ecurep/tmp/ITM_Health/Awn0gXM2ky/ -logpath /ecurep/pmr/4/8/48002,080,678/2017-05-24/48002.080.678.pdcollect-w94486.tar.Z_unpack/autopdzip/autopd/IBM/ITM/logs/logs w94486_ms_59243128
"working on 0 /ecurep/pmr/4/8/48002,080,678/2017-05-24/48002.080.678.pdcollect-w94486.tar.Z_unpack/autopdzip/autopd/IBM/ITM/logs/logs/w94486_ms_59243128-01.log"
The header is followed by advisory messages. These are indications of errors and warnings from the diagnostic log. The advisories are sorted in reverse order by impact. These impact values selected based on years of experience however feel free to use you own judgement. These are Advisory and not The Law. The following is just one recently created. There are 56 advisories at ths\is writing and more are added all the time.
Advisory Message Report - *NOTE* See advisory notes at report end
200,TEMSAUDIT1052E,HUB,Hub TEMS has lost connection to HUB 84 times
100,TEMSAUDIT1050W,PCB,Agent connection churning on  systems total - See following report
90,TEMSAUDIT1055W,Agent,Agent Location Flipping Changes Detected  - See following report
90,TEMSAUDIT1061E,TEMS,Situation [ddb_sanprb_x07c_aix_gen] with unknown attribute [K07_SANPATH.MonitoringSolution] - [*IF ( ( *VALUE K07_SANPATH.RC *EQ '20' ) *OR ( *VALUE K07_SANPATH.RC *EQ '21' ) *OR ( *VALUE K07_SANPATH.RC *EQ '22' ) *OR ( *VALUE K07_SANPATH.RC *EQ '23' ) ) *AND *VALUE K07_SANPATH.MonitoringSolution *EQ 'aix']
80,TEMSAUDIT1016W,Onlines,3 Agents with repeated onlines
80,TEMSAUDIT1035W,diagnostic,7 worrying diagnostic messages - see later report section
80,TEMSAUDIT1054W,SOAP,SOAP Errors Types  Detected - See following report
50,TEMSAUDIT1021W,timeout,4 Agent time out messages
0,TEMSAUDIT1005W,Stack,ulimit stack  is above nominal  (kbytes)
The pattern of each advisory is an impact level, an advisory identification, a category and a text message. Some reference later report sections. The first one above
200,TEMSAUDIT1052E,HUB,Hub TEMS has lost connection to HUB 84 times
Impact: 200 The highest current level.
ID: TEMSAUDIT1052E a sequential number
Category: HUB the hub TEMS is involved
Text: Hub TEMS has lost connection to HUB 84 times
At the end of the temsaud.csv report you find the following explanations of each advisory. Here is the one for the issue above.
Advisory code: TEMSAUDIT1052E
Text: Hub TEMS has lost connection to HUB count times
(58D77E39.0002-11A0:ko4mgtsk.cpp,133,"ManagedTask::sendStatus") Connection to HUB lost - stopping situation status insert
Meaning: This is produced when one hub TEMS task can no longer connect with the dataserver function of the hub TEMS. This is a severe error which is only recovered with a hub TEMS recycle.
The hub TEMS will not recover without a recycle.
This is a most severe issue and needs prompt diagnosis. It can be caused by many reasons including TEMS database problems, network problems and excessive workload.
Recovery plan: Contact IBM support for aid in diagnosis.
<end of advisory explanation>
Next follow a series of report sections. Some are always present others only when specific conditions are observed.
1) Results Summary
Trace: error (unit:kpxrpcrq,Entry="IRA_NCS_Sample" state er)
This presents the impact of incoming result data from agents. The “Total Results per minute” is probably most important. Experience shows that a rate of 500,000 bytes per minute is easily sustainable. Depending on system power and storage higher incoming rates can be accommodated. At 5megs/min is problems are usually seen. The highest ever measured was 127 megs/min and the TEMS was in a sorry state.
Total Result (bytes),,,12239004
Total Results per minute,,,31737
The trace report was added because one TEMS was seen with RAS1=ALL and the TEMS was just barely surviving.
KBB_RAS1= ERROR (UNIT:kfastinh,ENTRY:"KFA_InsertNodests" OUT ER) …
Trace duration (seconds),,,30938
Trace Lines Per Minute,,,178
Trace Bytes Per Minute,,,29473
The No Matching Request line shows severe TEMS stress. A real-time data request was made, the request timed out and later the agent returned data which was discarded of course. A well running TEMS never sees this issue.
Sample No Matching Request count,,,14,
Following is the report section that shows which situations or real time request are causing the most data to be returned. The largest contributors are shown first. This is a relatively lightly running system and HEARTBEATs [technically node status updates] dominate at 43% of the bytes. More often one or more situations totally dominate. By stopping or rethinking those the workload stress can be relieved. This is often not seen as high CPU. TEMS has a lot of internal locking and the effect is a lot of waiting and general slow processing.
Situation Summary ReportSituation,Table,Count,Rows,ResultBytes,Result/Min,Fraction,Cumulative%,MinResults,MaxResults,MaxNode
... more follow
Following is a second sorting of report that shows which managed systems are sending the most result data and what the peak contributing situation is.
Managed System Summary Report - non-HEARTBEAT situations
... more follow
2) Result Detail
Trace: error (unit:kpxrpcrq,Entry="IRA_NCS_Sample" state er)
Parameter added -rd
The first extra report shows a minute by minute presentation of incoming results and the top 5 contributors. This can be very helpful in understanding burst behavior. In one case a burst of 139 megs was arriving every 15 minutes which crushed remote TEMS processing – even though the long term average wasn’t so bad.
This linear report is followed by a graphical display. Here is an extract
Situation Result Over Time Graph - peak rate is 1303148 bytes per minute
Each hour is shown, each column is a minute, numbers represent 10 minutes
. . .
. . . .
.. .. .. ..
.. .. .. ..
.. .. .. ..
.. .. .. ..
Each column represents 10% of the maximum, rounded to nearest 10%. This pattern seems to show a burst roughly every fifteen minutes. In this case the detailed report looked like
The workload was not situation related but was likely driven by a TEPS workspace summary display against KVM servers. Peak rate was only 1.3 megs/minute and so probably sustainable.
3) Endpoint Communication Problem Report
1C010001:1DE0000F, Endpoint unresponsive,3,
This reports on ITM communications failures. In any large environment, there are usually some number of these. If there are a lot of them there may be network issues or the TEMS or the agent can be overloaded. In this case a hub TEMS saw time outs talking to one remote TEMS and that could mean an overloaded remote TEMS.
4) Reflex Command Summary Report
Trace:error (unit:kraafira,Entry="runAutomationCommand" all)(unit:kglhc1c all)
First section shows the action commands run and counts.
1651,0,1649,"/opt/IBM/ITM/scripts/bsm_history.pl ""MHC_COG_SU_Marketing_Tx"" 'RRT_Response_Time_Critical' 3 ""1170306151941000"" 5 ""Performance degraded"" >/dev/null"
The second part shows the maximum simultaneous number seen operating at one time. All commands run in the same process space as the TEMS and as subprocesses. There have been cases where hundreds ran at the same time and destabilized the TEMS.
Maximum action command overlay – 32
0,/opt/IBM/ITM/scripts/bsm_history.pl "MHC_SAP_SMP_Login_Tx" 'RRT_Response_Time_Critical' 3 "1170306152002000" 5 "Performance degraded" >/dev/null,
1,/opt/IBM/ITM/scripts/bsm_history.pl "MHC_SAP_SMP_Login_Tx" 'RRT_Response_Time_Critical' 3 "1170306151944000" 5 "Performance degraded" >/dev/null,
2,/opt/IBM/ITM/scripts/bsm_history.pl "MHC_COG_SU_Marketing_Tx" 'RRT_Response_Time_Critical' 3 "1170306151947000" 5 "Performance degraded" >/dev/null,
Action commands running at the time can be very intensive and can even trigger TEMS instability. Avoid that by limiting such usage. The highest intensity is when the action command is configured to run on each evaluation, not just the first time. That is rarely useful and should be avoided.
5) SQL Summary Report
Trace: error (unit:kdssqprs in metrics er)
1655,User=SRVR01 Net=ip.spipe:#188.8.131.52.,"SELECT NODE, THRUN
232,User=sufuser Net=ip.ssl:#184.108.40.206:52078.,"INSERT INTO O4SRV.TS
151,User=KSH Net=ip.ssl:#220.127.116.11:52072.,"SELECT SITNAME, ORIGINNO
80,User=SRVR01 Net=ip.spipe:#100.66.233.8.,"SELECT AFFINITIES,HO
75,User=KSH Net=ip.ssl:#18.104.22.168:52085.,"SELECT SITNAME, ORIGINNOD
This report shows the SQL being processed by the TEMS. If this is high it may imply a work overload at the TEMS.
6) SQL Detail Report
Trace: error (unit:kdssqprs in metrics er)
Parameter added -sqldetail
source,911,4993,10.95,User=SRVR01 Net=ip.spipe:#22.214.171.124., table,896,4993,10.77,,O4SRV.INODESTS,
sql,881,4993,10.59,,,SELECT NODE, THRUNODE, HOSTADDR FROM O4SRV.INODESTS sql,15,4431,0.20,,,SELECT ORIGINNODE, PRODUCT, O4ONLINE, THRUNODE,
This is a detailed report showing the source of SQL, the tables involved and the SQL statement instances. In this case the SQLs were being processed 32.90 times a minute. The source was probably another TEMS [based on the 3660 port]. The table involved was INODESTS or the in-core node status table.
This can reflect a work overload condition. In one case the high SQL came from agents that were supposed to have been taken out of service.
7) SOAP SQL Summary Report
Trace: error (unit:kshdhtp,Entry="getHeaderValue" all) (unit:kshreq,Entry="buildSQL" all)
ip.ssl:#127.0.0.1:41067,83,"SELECT NODE, AFFINITIES FROM O4SRV.TNODELST WHERE NODELIST='*HUB'"
ip.ssl:#127.0.0.1:41067,83,"SELECT VALUE FROM O4SRV.TSYSVAR WHERE NAME ='KT1_TEMS_SECURE'"
ip.ssl:#127.0.0.1:41067,39,"SELECT NODELIST FROM O4SRV.TNODELST WHERE NODE='REMOTE_uswhram022hasra' AND NODETYPE='M'"
ip.ssl:#127.0.0.1:41067,39,"SELECT THRUNODE, AFFINITIES FROM O4SRV.INODESTS WHERE NODE='REMOTE_uswhram022hasra' "
ip.ssl:#127.0.0.1:59724,37,"SELECT THRUNODE, AFFINITIES, VERSION, O4ONLINE FROM O4SRV.INODESTS WHERE NODE='has_D219421VCSS0001: NT'"
This shows the SQLs coming through SOAP and the origin. 127.0.0.1 means it was run on the hub TEMS itself. These are often tacmd functions which use SOAP for many functions. There have been cases where so many tacmd functions were run that the hub TEMS became unstable. Thus caution should be observed.
8) Process Table Report
Trace: error (unit:kdsstc1,Entry="ProcessTable" all er)
Process Table Duration: 92 seconds
This report summarizes the completion of each SQL process. If the numbers of a table is very high that can result can indicate a overload work condition. The resolution is to reduce the workload or run on a more powerful system.
9) PostEvent Report
Trace: error (unit:kfastpst,Entry="KFA_PostEvent" all er)
This is seen at the hub TEMS and it shows the number of events arriving. If the numbers are very high this can severely impact the hub TEMS and the TEPS. Situations should be rare and exceptional reports and not arrive in floods.
10) Multiple Agent Online Report
The diagnostic log shows that agents were coming online over and over. That often suggests that different systems are running agents with the same name. That is a problem since ITM expects to have unique names. It also means that only one agent at a time is being monitored. Also the condition can cause TEMS instability. Usually you need IBM Support to track down the duplications.
11) Invalid Node Name Report
VM::v5wvcs01-a0001peenxg0001:ESX,1, Validation for affinity failed.,
VM:v5wvcs01-a001p5eenxg0002:ESX ,1, Validation for affinity failed.,
VM:v5wvcs03-a00001p5eenxg0006:EX,2, Validation for affinity failed.,
PRR-ContabilitÃ :Grp ,1, Validation for node failed.,
Nodes and nodelists can be invalid for several reasons. In some cases illegal characters are used in others [like this case] application support is missing. From ITM 630 on such agents are rejected by default and so monitoring is not being performed as expected. Names should be changed to legal names and application support should be added to increase the quality of monitoring.
12) Reflex [Action] Command failures
An intended action command failed. This needs to be researched otherwise the expected command does not run. The Status is platform dependent. The 4 usually means an exception or crash.
13) Fast Simple Heartbeat report
Trace: error (UNIT:kfaprpst ER ST)
The goal here is to show the inter-arrival time of agent node status updates. The first one shows must 600 seconds [10 minutes] but there were 3 at 33 seconds early and 3 at 33 seconds late. If there is a tremendous variability, that suggests that the TEMS is overloaded or a network problem. TEMS is a real-time system in many ways and so it should be run with enough capacity to handle the ebbs and flows of activity.
An overloaded TEMS does not usually show as high CPU. There is a lot of internal locking and the result is usually just a slowdown of normal processes.
14) Major Jitter Report
Trace: error (UNIT:kfaprpst ER ST)
This reports the second of the minute when agents send status which are far away from the expected regular timing.
15) Send Node Status Exception Report
Trace: error (UNIT:kfastinh,ENTRY:"KFA_InsertNodests" ALL)
This records a time when a remote TEMS sends an updated agent status to a hub TEMS. If this occurs a lot that may indicate duplicate agents or some other agent misconfiguration or perhaps a network issues.
16) Agent Timeout Report
This relates to a TEMS attempting to communicate with an agent concerning a situation. That usually happens when the agent has registered with the TEMS but is not ready for full communications. To understand what agent is involved you needed added kpx tracing to see the agent address and timing and context.
17) RPC Error report
This reports on Remote Procedure Call errors. These can indicate network or TEMS or Agent workload issues.
18) Historical Export summary by time
Trace: error (unit:khdxdacl,Entry="routeExportRequest" state er)
(unit:khdxdacl,Entry=" routeData" detail er)
This trace setting also applies to the next two report sections
For large environments it is best practice to collect data at the agents and export to the WPA from the agents. However if you do collect at the TEMS and export, this tells you how much data is being exported over time. At one customer, the import rate was 25 megs/min and the export rate was 10 megs/minute – largely because of network limitations. This report helped them make better choices about how much to collect.
19) Historical Export summary by object
How much export data by attribute group.
20) Historical Export summary by Object and time
How much export data by attribute group over time.
*Note: You can get the same sort of data from an ITM agent using the following trace
Trace: error (unit:khdxcpub,Entry="KHD_ValidateHistoryFile" state er) (unit:khdxhist,Entry="openMetaFile" state er) (unit:khdxhist,Entry="open" state er) (unit:khdxhist,Entry="close" state er) (unit:khdxhist,Entry="copyHistoryFile" state er)
21) Time Slot Result workload
Obsolete - Replaced by result detail.
22) Attribute File Warning Report
attribute name conflict,Diagnostic.IMPORTANCE,
attribute name conflict,Diagnostic.MESSAGE,
TEMS depends on accurate consistent application support files including the attribute files. If an attribute is multiply defined, you will get messages like. The above case is simple and no trouble – although there is an APAR to eliminate. Most common are cases where the previous attribute file has been saved under a different name… like ktu.atr.orig. TEMS processes all names and not just the .atr names. The solution there is just to erase the saved file, or move it to another directory. In other cases contact IBM Support to achieve resolution.
The impact is that some attributes might be mis-understood and situations not work as expected.
23) Loci Count Report
Loci Count Report - 25811 found
This is a catch-all report. The theory is that error messages often show is great volume. Some are ignored as known and uninteresting. The remaining ones are displayed. The first section or locus is the source unit, the function name and the line number. The next is a count of lines and a percentage of total recorded lines. In this way uncategorized error messages *may* be displayed for analysis. When the message count is very small the report is relatively un-interesting.
24) Agent connection churning Report
Agent connection churning Report - 3 systems
In a well running ITM environment connections are made [NewPCB] and then stay active for long periods of time – weeks or months. Sometimes we see the same ip address being created and then deleted over and over. That often means a configuration issue at the agent, where two agents are contending for the same connection. Often a mis-use of KDEB_INTERFACELIST causes the issue. In any case, the affect agents are severely affected and the TEMS itself can experience serious issues include failure at high rates. This report is of only 3 systems so the big effect is that the agents involved are likely not monitoring as expected.
25) SOAP Error Report
8,Unable to open request (79),
,8 , ip.ssl:#127.0.0.1,
This shows SOAP problems. Usually they are seen during development periods. However they can also be seen when there are too much simultaneous SOAP activity. Usually the hub TEMS stability is not affected.
26) Agent Flipping Report
Parameter added -flip
This report is new at ITM 630 FP7 on a hub or remote TEMS. It means that a given agent is reporting differently. In the first set of 3 a Windows OS Agent Primary:ATENEA2:NT is reporting from two different ip addresses. In addition one of the addresses 10.231.33.95 is reporting through two different remote TEMSes.
In the second set one agent is reporting through two different remote TEMSes, apparently flipping back and forth. That is true for two different agents.
In the report, there were 1678 lines in that report section. Things are seriously wrong.
There can be many reasons. For example, the agents may be configured with the same name even though they are on different systems. If the OS agent is in the ITM 622 GA to FP2 level, the agent can be connected to two different remote TEMS at the same time. And there are lots of other potential problems. The impact is severe. You can wind up running situations on only part of the agents and the TEMS itself can be unstable and crash.
In a case like this you usually need to work with IBM support to resolve this issue.
27) Situation Length 32 Report
This is rarely seen. It usually means a situation was constructed manually and uploaded using tacmd createsit without the benefit of the TEP situation editor validity checking. The result is that the situation is not working, which is obviously a poor result.
On rare occasions this might be a leftover from a ITM 6.1 created situation where 32 character names were legal.
The resolution is to re-author the situations in the TEP situation editor.
Tracing Guide in a Separate Document - click here
The tracing appendix is in a separate post because of a document size limit. It shows exactly how to set diagnostic traces for TEMS Audit both statically and dynamically.
This document shows how to collect performance data from a TEMS. As a result you can make operations more efficient and complete.
Here are recently published versions, In case there is a problem at one level you can always back up.
A year of changes, too many to list here. See the temsaud.pl history section if interested.
Restructure for improved standalone running. Add several new advisories low nofiles and concurrent action commands
restructure advisory messages, add advisories for TEMS database errors
Correct some logic on incoming PostEvent messages
Handle a null SQL text continuation better
show more contributors to the resource interval report
Show duplicate agent online better - top 20
Add SimpleHeartbeat report - finds another type of duplicate agent
Add Major Jitter correlation report
Make command capture logic work on z/OS logs
Add result interval report
Add results interval count report
Restore ProcessTable report section
Add PostEvent report section
Add SQL report section and a maximum concurrent action command section
Add action command report section
Correct defect on z/OS log and tracking listen pipes
Add Soap Burst Advisory and SOAP Detail report
Add advisory for ulimit stack more then 10M
ProcessTable Summary, listen pipes. "No Matching Request" error summarized, Nofile advisory, improve -z option processing
1.10000 - last technote version
Advisory section. 16meg truncation warning
Identify and correct workload and configuration problems. I encourage anyone to share success stories, enhancement requests or problems found.
Note: Art Deco Cat sculpture