Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
6 replies Latest Post - ‏2012-04-11T23:25:39Z by Guus.Bonnes
SystemAdmin
SystemAdmin
310 Posts
ACCEPTED ANSWER

Pinned topic Any correlation between interval file size, numbufs and bufsize parameters?

‏2012-03-29T16:54:00Z |
My general question about Access Monitor, is if I make the interval file bigger does that mean I can make numbufs and bufsize smaller? The way I'm thinking of it is that the interval file being the disk cachce for Access Monitor the bigger that is the smaller the buffers need to be (or less bufffers there need to be). Is it that too simplistic of a way to think about how those parameters work though?

I'd like to ideally make the interval file larger I would think then because there are limits to the number of buffers that can be had and their size.

Thanks,
Joel
Updated on 2012-04-11T23:25:39Z at 2012-04-11T23:25:39Z by Guus.Bonnes
  • Guus.Bonnes
    Guus.Bonnes
    161 Posts
    ACCEPTED ANSWER

    Re: Any correlation between interval file size, numbufs and bufsize parameters?

    ‏2012-03-30T07:47:56Z  in response to SystemAdmin
    I'm not quite sure what you mean by interval file... let me just ramble a bit about how the process works, and maybe that answers your question.

    Access Monitor intercepts all relevant RACF events. It collects information about these events in a record. These records are moved to an in-storage buffer. Every minute (default), a buffer switch occurs. The old buffer is read and processed by CKRCARLA and counted/summarized into working storage of the CKRCARLA program. Every half hour (default) CKRCARLA writes out (disp=mod) all summarized data to a disk data set (the daily collection data set). Sometimes there are too many events to fit into the one-minute buffer. In that case we do an extra intermediate switch to the next buffer. That extra buffer is not immediately processed, but kept in storage till the end of the minute. So every minute, we fill and read/process one or more in-storage buffers.
    Ideally, the event rate is constant over the day, so we can exactly match the buffer size to be just large enough for one minute worth of events. And then we only need three buffers (one to collect in, one being read from, and one being switched to). But alas, event rates fluctuate. So, we specify more buffers, which can be somewhat smaller. Now our target is that during the slack periods, a single buffer is still large enough to contain all events for one minute, but that during the peak periods, we use about one third of the available buffers.
    The disk data set used to store the half-hour of collected data (the daily collection data set) grows over the day. At the end of the day it has a concatenation of 48 collections of half-hour data. We close the data set and start a new one. A new subtask is started, that processes the disk data set and which consolidates the 48 half-hour data into a single 24 hour data set (the daily consolidation data set).
    If the C2PACMON started task is stopped/started during the day, we don't use the same data set as we used last time. We always create a new one. To come-up with a unique name, we use a timestamp in the data set name. During normal operation, there is only one such data set per day. There are alternatives, like creating a new collection data set every half-hour. But the current product doesn't do that, and uses only one daily collection data set per start/stop of the started task.

    To answer what I perceive as the core of the question that you asked: the size of the data set on disk, and the number/size of the buffers specified both depend on the number of events during the day. You can juggle with the number/size of the buffers, but the end result needs to be that you have enough space for three minutes' worth of data. And the end result for the data set on disk is that it always needs to be the right size for one day's worth of data.

    Hope all this helps...

    Guus Bonnes
    zSecure Access Monitor architect and designer
    • Don.Hoben
      Don.Hoben
      2 Posts
      ACCEPTED ANSWER

      Re: Any correlation between interval file size, numbufs and bufsize parameters?

      ‏2012-04-11T16:45:23Z  in response to Guus.Bonnes
      Pardon the interruption, but I'm struggling with the same problem and was referred to this thread for the solution.

      As I understand what Guus has described, the size of the file has no bearing on the error message. Only adjusting the bufsize and bufnum as indicated in the C2PAMP member will have an effect.

      The default parms as delivered were:
      option bufsize(64) numbufs(8)

      If this is being written to the ".X." file it seems too small from the start. The record size for that file is VB 556. How does that match up with the default values?

      I see there are some parms on the DEBUG keyword that may help pin this down. The doc indicates the stats will be written to the consold as well as the job/syslogs. I'm a little bit leary of writing to the console, our operators may get riled up, but this seems a logical step in setting the buffer parms correctly.

      Thoughts?

      Don
      • SystemAdmin
        SystemAdmin
        310 Posts
        ACCEPTED ANSWER

        Re: Any correlation between interval file size, numbufs and bufsize parameters?

        ‏2012-04-11T18:26:12Z  in response to Don.Hoben
        Hi Don,
        Not a problem. Happy to help.
        Let me ramble about some of experiences in order to help answer your question.

        I've been running Access Monitor for almost a year now and been through numerous history loss messages. In the past when the occured we simply incrased the bufsize and numbfus then restarted the consolidation process (f c2pacmon,consolidate).

        I'm currently at option bufsize(8192) numbufs(16) on our sandbox. That's right a little sandbox lpar that has barely any people logging in (10 tops), full compliement of STCs for z/OS, CICS, IMS, MQ and numerous ISV products installed got up to that much buffer size. Why you say?

        Well we only increased the bufsize and numbufs when we saw loss of history messages. Until Guus's reply (which I'll respond to hopefully after I eat lunch as I'm getting hungry) I wasn't truly sure if the size of the data collection file (the file that C2PACMON allocates upon start up) was also a factor. That is the way I read Guus's post is that the size of the data collection file (which is specified in member C2PAMCLT with a default of 1,1 cylinders) is a factor in how many and how much buffer you need. The larger I make the data collection (or temporary work file) that the task uses to write out collected data then theoretically the less buffers I should need.

        Now I've left our c2pamp on our sandbox LPAR set to:option bufsize(8192) numbufs(16)
        report interval(60)
        report member(c2pamcol)
        report consolidatemember(c2pamcon)
        report consolidatetime(0000)

        For the the data collection and consolidation file we've decided to increase to 100,25 cyl and so far we have yet to see any more messages about C2PACMON's buffers being over run with data.

        How many buffers and how big they are is a function of how much SAF activity you have. So it is hard to estimate. It's a trial and error process where you just have to keep increase the size of the buffers or the number of them OR the size of the daily collection file so that you make sure you aren't losing the information that C2PACMON is collecting from three different exist points.

        So I avoided using DEBUG and this trial & error process worked great.

        So you don't have to completely stop and start the task for a simple change to C2PAMP but if you change the size of the daily collection or consolidation file then you'll have to do that but note there is a RESTART option but we have our C2PACMON under the control of automations so using f c2pacmon,restart would not be desired.

        I wrote that rather quick off the top of my head pretty much so please excuse any typos.
        (Too hungry to spell check!)

        My two cents. Your mileage may very. Hope that helps,
        Joel
        • Guus.Bonnes
          Guus.Bonnes
          161 Posts
          ACCEPTED ANSWER

          Re: Any correlation between interval file size, numbufs and bufsize parameters?

          ‏2012-04-11T23:25:39Z  in response to SystemAdmin
          Joel,

          > ... that the size of the data collection file
          > (which is specified in member C2PAMCLT with a default of
          > 1,1 cylinders) is a factor in how many and how much
          > buffer you need.

          Not quite. Both sizes are dependent on the number of events that occur. You cannot trade one for the other. If there are many events, you need more buffers and you need larger data sets.

          If you change the data set allocations you can either use a F C2PACMON,CONS or a F C2PACMON,REST command. If you change the buffer specification, you'll need the F C2PACMON,REST command. When using the RESTART command, there is a short period that events are not captured. With the CONSOLIDATE command, all events continue to be captured.

          Guus Bonnes
      • Guus.Bonnes
        Guus.Bonnes
        161 Posts
        ACCEPTED ANSWER

        Re: Any correlation between interval file size, numbufs and bufsize parameters?

        ‏2012-04-11T23:04:47Z  in response to Don.Hoben
        Don,

        Indeed, the C2P warning or error messages about buffers (history data lost, or recent data lost) have no relation whatsoever to the size of the disk data sets. If the disk data set is too small, you will get an ordinary E37 (oslt). The subtask abends, a message appears on the console, and somebody needs to investigate and issue a F C2PACMON,RESTART.

        The default number of buffers and the buffer size is probably a bit small. 8 buffers of 64K is enough for me on my single user z/OS. On the other hand I have not heard of any customer where 16 buffers of 1M didn't suffice. We have some customers with >200M events per day.

        The default size of the data set where all these records are saved is probably also a bit small. The data set is VB(556), but most of the time the records are around 60 chars. That's where the blocksize comes into play. You'd probably want dasd optimum blocksize, as determined by dfsms.

        The DEBUG BUF command can be used to generate WTO messages about buffer usage. This results in two WTO messages every minute. If you have IMS or MQ on your system, these two extra message are probably insignificant. The messages show how many bytes are used in the buffer(s) for the last minute. It can help you determine if the buffer is large enough, and if you have enough buffers. As I noted in my previous append, you'd want to use about a third of the total available space. I have a preference for more, but somewhat smaller buffers. As a side note, we are talking of somewhere around a total of 16Mbyte buffer space in the private area of one started task. I can imagine that some people might just specify the max, and forget about it.

        Oh, and one more thing, the RESTART command performs an internal restart of all the processes. The C2PACMON started task itself stays up and running. So depending on what your automation process triggers on, it might not even see the restart happening.

        Regards,

        Guus Bonnes
        • Guus.Bonnes
          Guus.Bonnes
          161 Posts
          ACCEPTED ANSWER

          Re: Any correlation between interval file size, numbufs and bufsize parameters?

          ‏2012-04-11T23:15:18Z  in response to Guus.Bonnes
          > ... we are talking of somewhere around a total of 16Mbyte buffer space in the private area of one started task...

          Remember: "Always reread before you press Post".
          I should have said "32 buffers of max 16Mbyte buffers".

          It does add up, but it's all pageable virtual storage.

          Regards,

          Guus Bonnes