Topic
  • 4 replies
  • Latest Post - ‏2012-12-11T11:25:18Z by SystemAdmin
SystemAdmin
SystemAdmin
4179 Posts

Pinned topic Performance Monitoring APIs?

‏2012-12-08T19:09:57Z |
We are interested in capturing Message Flow Accounting and Statistics Data as well as Resource Statistics programatically.

It seems that it is possible to:
Generate XML documents that provide information on Resource Statistics.
Generate XML documents describing message flow events.

We are concerned about the overhead in parsing and processing these documents at high volume. Is there an API to obtain the same detailed information? The API documentation I've seen seems to allow monitoring to be configured but I don't see access to the detailed output.

Any pointers gratefully received.

Thanks

Ben
Updated on 2012-12-11T11:25:18Z at 2012-12-11T11:25:18Z by SystemAdmin
  • antonpiatek
    antonpiatek
    31 Posts

    Re: Performance Monitoring APIs?

    ‏2012-12-10T08:15:08Z  
    Both the resource and flow statistics are emitted every 20 seconds. A ball-park cost for turning these on is something like 5-10% throughput redcution.

    I wouldn't expect parsing a few xml documents every 20 seconds to be particularly taxing. I don't believe there is any other way to get statistics data (this is how the Message Broker Explorer statistics page does it)
  • SystemAdmin
    SystemAdmin
    4179 Posts

    Re: Performance Monitoring APIs?

    ‏2012-12-11T00:07:14Z  
    Both the resource and flow statistics are emitted every 20 seconds. A ball-park cost for turning these on is something like 5-10% throughput redcution.

    I wouldn't expect parsing a few xml documents every 20 seconds to be particularly taxing. I don't believe there is any other way to get statistics data (this is how the Message Broker Explorer statistics page does it)
    Thanks Anton. It is a bank with an expected ~700,000 messages a day or 16 per second on average over a 12 hour day. The intention is to monitor flows automatically using external monitoring software (which will read and parse the files) to see that they are being processed in a timely manner and if there is a delay the next step would be to review the resource statistics to try and identify where the problems is. We will try and do the monitoring off-board so as not to impact the AIX LPAR running WMB at least for the file parsing. If we were to turn on flow statistics for everything there would be a lot of files to parse I would have thought. We are very new to WMB so we might be missing something and frankly we might be going down an ill-advised path. Does this description of what we are trying to do sound sensible, normal practice, or crazy? Ben
  • antonpiatek
    antonpiatek
    31 Posts

    Re: Performance Monitoring APIs?

    ‏2012-12-11T08:31:15Z  
    Thanks Anton. It is a bank with an expected ~700,000 messages a day or 16 per second on average over a 12 hour day. The intention is to monitor flows automatically using external monitoring software (which will read and parse the files) to see that they are being processed in a timely manner and if there is a delay the next step would be to review the resource statistics to try and identify where the problems is. We will try and do the monitoring off-board so as not to impact the AIX LPAR running WMB at least for the file parsing. If we were to turn on flow statistics for everything there would be a lot of files to parse I would have thought. We are very new to WMB so we might be missing something and frankly we might be going down an ill-advised path. Does this description of what we are trying to do sound sensible, normal practice, or crazy? Ben
    There are two things which I think you need to separate in your mind.
    • Accounting and Statistics - this emits a message per flow and/or per "resource manager" every 20 seconds, and contains statistics like how many messages were processed, processing time, elapsed time, etc.
    • Monitoring - This emits a message from every configured node event, you may choose to have one per flow, multiple per flow, or only on certain triggers. These are useful for tracking what has been processed and may be used with something like the Record and Replay function new in v8.

    It isn't clear exactly which you need from your description. I don't think there is a pre-canned metric for how long a flow took to process a message, however you might find that taking the Accounting and Statistics data and using elapsedTime/numberOfMessagesProcessed that you get a pretty good elapsed time per message. This is then only 3 messages a minute to process on your external monitoring (per flow).
    Accounting and Statistics also allows a second type of statistic tracking (the one above I talk about is Snapshot statistics), Archive Statistics allow a user-defined time interval (in minutes I believe) so you could use this to track the same data but over a period of 5, 10 or even 60 minutes, which would reduce the workload on your other systems.
    I would start by reading about the Accounting and Statistics function in WMB: http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/topic/com.ibm.etools.mft.doc/ac18920_.htm

    Resource statistics are similar, but are tracked from the point-of-view of WMB resource managers, such as CICS connections, file/ftp handles, jmv, jdbc, odbc connections, etc See http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/topic/com.ibm.etools.mft.doc/bn43250_.htm
  • SystemAdmin
    SystemAdmin
    4179 Posts

    Re: Performance Monitoring APIs?

    ‏2012-12-11T11:25:18Z  
    There are two things which I think you need to separate in your mind.
    • Accounting and Statistics - this emits a message per flow and/or per "resource manager" every 20 seconds, and contains statistics like how many messages were processed, processing time, elapsed time, etc.
    • Monitoring - This emits a message from every configured node event, you may choose to have one per flow, multiple per flow, or only on certain triggers. These are useful for tracking what has been processed and may be used with something like the Record and Replay function new in v8.

    It isn't clear exactly which you need from your description. I don't think there is a pre-canned metric for how long a flow took to process a message, however you might find that taking the Accounting and Statistics data and using elapsedTime/numberOfMessagesProcessed that you get a pretty good elapsed time per message. This is then only 3 messages a minute to process on your external monitoring (per flow).
    Accounting and Statistics also allows a second type of statistic tracking (the one above I talk about is Snapshot statistics), Archive Statistics allow a user-defined time interval (in minutes I believe) so you could use this to track the same data but over a period of 5, 10 or even 60 minutes, which would reduce the workload on your other systems.
    I would start by reading about the Accounting and Statistics function in WMB: http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/topic/com.ibm.etools.mft.doc/ac18920_.htm

    Resource statistics are similar, but are tracked from the point-of-view of WMB resource managers, such as CICS connections, file/ftp handles, jmv, jdbc, odbc connections, etc See http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/topic/com.ibm.etools.mft.doc/bn43250_.htm
    Thanks Anton for taking the time to respond. I'm going to do some more reading and set up a system with the samples and see what output pops out and go from there. Much appreciated. Ben