Monitoring agent is running out of storage

If response time is slow on the monitoring agent, check the RKLVLOG for messages about a possible storage problem. A storage usage event is generated every hour and written to the RKLVLOG. This behavior is driven by a scheduling command in RKANCMDU member KDSSTART.

The KDSSTART member of RKANCMD has the following line by default.

EVERY 00:60:00 STORAGE D * LOG STORAGE USE

Where

Value Description
EVERY Is a command that is used to schedule another command for periodic execution.
00:60:00 Shows the hours, minutes, and second. For example, a value of 00:60:00 says run this command every 60 minutes.
Storage Displays statistics of ITMS:Engine storage usage.
D Stands for Detail. STORAGE provides both primary and extended storage statistics. This information is helpful in tuning ITMS:Engine memory management.

The default entry causes the STORAGE D command to be issued every 60 minutes. The interval at which the command is automatically issued is defined during the Tivoli® Enterprise Monitoring Server configuration with the Enable storage detail logging parameter on the Specify Advanced Configuration Options panel, where the values are Y or N. There are two other settings associated with the Enable storage detail logging parameter:

  • Set the Storage detail logging interval to monitor storage. The interval values are written as part of the second EVERY command in &rhilev.&midlev.RKANCMDU(KDSSTART). The default is 60 minutes.

  • Set the Flush VSAM buffers interval to force all deferred VSAM writes to DASD. The interval values are written as part of the third EVERY command in &rhilev.&midlev.RKANCMDU(KDSSTART). The default is 30 minutes.

  • To disable storage detail logging, set this parameter to N, which then generates the second EVERY command as a comment.

To control storage detail logging dynamically, you can issue the following command from the z/OS console to the monitoring server started task to enable storage detail logging:

MODIFY procname,STORAGE D

Where:

Value Description
Modify Is the z/OS operator command.
procname Is the name of the monitoring server started task. The default name of monitoring server started task in CANSDSST.
STORAGE Displays statistics of ITMS:Engine storage usage.
D Stands for Detail. STORAGE provides both primary and extended storage statistics. This information is helpful in tuning ITMS:Engine memory management.

After you issue this command, look in the RKLVLOG for the output. See the following sample output of this command. Explanations are provided after the sample output.

01 KLVSD002 EXTENDED MAIN STORAGE INFORMATION:
02 KLVSD003    ALLOCATION DETAIL:
03 KLVSD004       SIZE(1-16) USE(414) TOTAL(4582) ACCESSED(5162)
04 KLVSD004       SIZE(17-32) USE(799) TOTAL(800) ACCESSED(1428)
05 KLVSD004       SIZE(33-48) USE(226) TOTAL(1606) ACCESSED(2061)
06 KLVSD004       SIZE(49-64) USE(21) TOTAL(22) ACCESSED(44)
07 KLVSD004       SIZE(65-80) USE(9) TOTAL(10) ACCESSED(30)
08 KLVSD004       SIZE(81-96) USE(16) TOTAL(16) ACCESSED(16)
09 KLVSD004       SIZE(97-112) USE(197) TOTAL(197) ACCESSED(197)
10 KLVSD004       SIZE(113-128) USE(40) TOTAL(40) ACCESSED(240)
11 KLVSD004       SIZE(129-144) USE(0) TOTAL(1) ACCESSED(2)
12 KLVSD004       SIZE(145-160) USE(1) TOTAL(1) ACCESSED(1)
13 KLVSD004       SIZE(161-176) USE(1) TOTAL(1) ACCESSED(2)
14 KLVSD004       SIZE(177-192) USE(1) TOTAL(1) ACCESSED(1)
15 KLVSD004       SIZE(193-208) USE(0) TOTAL(0) ACCESSED(0)
16 KLVSD004       SIZE(209-224) USE(0) TOTAL(5) ACCESSED(5)
17 KLVSD004       SIZE(225-240) USE(0) TOTAL(6) ACCESSED(7)
18 KLVSD004       SIZE(241-256) USE(10) TOTAL(13) ACCESSED(126)
19 KLVSD004       SIZE(257-288) USE(2) TOTAL(6) ACCESSED(9)
20 KLVSD004       SIZE(289-320) USE(0) TOTAL(1) ACCESSED(1)
21 KLVSD004       SIZE(321-352) USE(1) TOTAL(1) ACCESSED(1)
22 KLVSD004       SIZE(353-384) USE(2) TOTAL(2) ACCESSED(2)
23 KLVSD004       SIZE(385-416) USE(1) TOTAL(1) ACCESSED(1)
24 KLVSD004       SIZE(417-448) USE(0) TOTAL(0) ACCESSED(0)
25 KLVSD004       SIZE(449-480) USE(0) TOTAL(2) ACCESSED(2)
26 KLVSD004       SIZE(481-512) USE(5) TOTAL(7) ACCESSED(7)
27 KLVSD004       SIZE(513-576) USE(0) TOTAL(0) ACCESSED(0)
28 KLVSD004       SIZE(577-640) USE(1) TOTAL(4) ACCESSED(4)
29 KLVSD004       SIZE(641-704) USE(1) TOTAL(2) ACCESSED(3)
30 KLVSD004       SIZE(705-768) USE(0) TOTAL(2) ACCESSED(2)
31 KLVSD004       SIZE(769-896) USE(1) TOTAL(1) ACCESSED(1)
32 KLVSD004       SIZE(897-1024) USE(1) TOTAL(4) ACCESSED(8)
33 KLVSD004       SIZE(1025-1280) USE(41) TOTAL(41) ACCESSED(1042)
34 KLVSD004       SIZE(1281-1536) USE(0) TOTAL(1) ACCESSED(11)
35 KLVSD004       SIZE(1537-2048) USE(3) TOTAL(4) ACCESSED(5)
36 KLVSD004       SIZE(2049-4096) USE(9) TOTAL(10) ACCESSED(14)
37 KLVSD004       SIZE(4097-8192) USE(7) TOTAL(10) ACCESSED(357)
38 KLVSD004       SIZE(8193-16384) USE(7) TOTAL(7) ACCESSED(151)
39 KLVSD004       SIZE(16385-32768) USE(0) TOTAL(1) ACCESSED(1)
40 KLVSD004       SIZE(32769-65536) USE(5) TOTAL(5) ACCESSED(7)
41 KLVSD004       SIZE(65537-131072) USE(1) TOTAL(1) ACCESSED(1)
42 KLVSD004       SIZE(131073-262144) USE(0) TOTAL(0) ACCESSED(0)
43 KLVSD004       SIZE(262145-524288) USE(1) TOTAL(1) ACCESSED(1)
44 KLVSD004       SIZE(524289-1048576) USE(1) TOTAL(1) ACCESSED(1)
45 KLVSD004       SIZE(1048577-2097152) USE(0) TOTAL(0) ACCESSED(0)
46 KLVSD004       SIZE(2097153-4194304) USE(0) TOTAL(0) ACCESSED(0)
47 KLVSD004       SIZE(4194305-8388608) USE(0) TOTAL(0) ACCESSED(0)
48 KLVSD005     LIMIT(8388608) SLOPE(15) SIZES(45) TOTAL(403319K)
49 KLVSD006     FREE(400928K) CARVED(2598K) OVERHEAD(59640)
50 KLVSD007   1% IS IN USE; 90% ALLOWED
51 KLVSD008   1% HAS BEEN CARVED; 95% ALLOWED
52 KLVSD021 TMS(0) PREFIX(8) CUSHION(8)
53 KLVSD031 BUFFER POOL INFORMATION
54 KLVSD032   POOL BUFSIZE(3564) SEGSIZE(65536) MASK(3FFFF) SIDEQ(0)
55 KLVSD033     BUFFERS INUSE(0) MAX(0) GETS(0) FREES(0)
56 KLVSD034     SEGMENTS INUSE(0) MAX(0) GETS(0) FREES(0) Q(0) QMAX(0)
57 KLVSD032   POOL BUFSIZE(3440) SEGSIZE(65536) MASK(7FFFF) SIDEQ(0)
58 KLVSD033     BUFFERS INUSE(0) MAX(0) GETS(0) FREES(0)
59 KLVSD034     SEGMENTS INUSE(0) MAX(0) GETS(0) FREES(0) Q(0) QMAX(0)
60 KLVSD032   POOL BUFSIZE(2560) SEGSIZE(65536) MASK(1FFFFFF) SIDEQ(0)
61 KLVSD033     BUFFERS INUSE(0) MAX(0) GETS(0) FREES(0)
62 KLVSD034     SEGMENTS INUSE(0) MAX(0) GETS(0) FREES(0) Q(0) QMAX(0)
63 KLVSD032   POOL BUFSIZE(1920) SEGSIZE(65536) MASK(FFFFFFFF) SIDEQ(0)
64 KLVSD033     BUFFERS INUSE(0) MAX(0) GETS(0) FREES(0)
65 KLVSD034     SEGMENTS INUSE(0) MAX(0) GETS(0) FREES(0) Q(0) QMAX(0)
66 KLVSD039 END OF BUFFER POOL INFORMATION

Where:

  • 01 is an ITMS:Engine header message.
  • 02 is an ITMS:Engine header message.
  • 03 to 47 is an ITMS:Engine message specifying the following values:
    • SIZE: The range (m-n, in bytes) of the sizes of data blocks in the storage area. For example, SIZE(1-16) indicates that this area contains all of the blocks that are from 1 to 16 bytes long.
    • USE: The number of blocks in use.
    • TOTAL: The total number of storage blocks allocated.
    • ACCESSED: The total number of times storage blocks in this range were accessed.
    Note: If the values of both USE and TOTAL are zero, the message is not displayed.
  • 48 is an ITMS:Engine message specifying the following values:
    • LIMIT: The size (in bytes) of the largest block that can be allocated.
    • SLOPE: An IBM-internal parameter.
    • SIZES: Specifies the number of storage areas.
    • TOTAL: Specifies (in kilobytes) the total amount of storage.
    • FREE: Specifies (in kilobytes) the amount of storage available.
    • CARVED: Specifies the amount of storage that has been carved into specific blocks for allocation.
    • OVERHEAD: The amount of storage (in bytes) used for storage control.
  • 50 is an ITMS:Engine message specifying the following values:
    • x% IS IN USE: Indicates that percentage of allocated storage that is currently in use.
    • x% ALLOWED: When the previous percentage listed reaches this value, storage allocation quiesces nuntil enough storage is freed to bring the IN USE percentage below this value.
  • 51 is an ITMS:Engine message specifying the following values:
    • x% HAS BEEN CARVED: Indicates the percentage of allocated storage that is currently carved into blocks for allocation.
    • x% ALLOWED: The maximum amount of allocated storage which can be carved into allocatable blocks.