MartinPacker 11000094DH Comments (8) Visits (3800)
In a sense this post follows on from I Must Be Mad - where I talked about some of the subtleties of processing SMF. In another sense it’s writing down in public a briefing I want to give one of my mentees.1
She’s about to prototype some code to go against a record subtype we’ve not handled before. In fact we throw any records of this subtype away - today.
… all the way back to a z/OS 2.1 presentation my friend and co-conspirator Marna Walle gave in 2013. (I had to look that one up.)
In it she mentioned z/OS TSO REXX being able to process Variable Blocked Spanned (VBS) data. The best use case for this is, of course, SMF.
I was delighted to see this support. But I’ve done nothing with it until now.
Actually I have some REXX code that processes SMF but I don’t really like it: It copies SMF 70-1 to a VB (not Spanned) data set and then a REXX exec processes this data. There’s a very good reason for not liking this: It risks truncating records. SMF 70-1 records can be very long, so this is a potentially serious problem.
So, processing VBS would avoid the VBS-to-VB copy and eliminate the risk of breakage.
The rest of this post is about some of the coding techniques for processing SMF with REXX. I actually developed them while processing SMF in VB format, but they are equally valid with VBS.
Processing SMF With REXX
While REXX is reasonably fast, I wouldn’t use it for high volume record types. My use case was extracting the LPARs on a machine which are deactivated. You get this from SMF 70-1 where the Logical Partition Data Section says there are no Logical Processor Data Sections.
This is a low volume case - as 70-1 records are only generated on an RMF interval and there are relatively few of them. My code processes 70-1 in a second or two.
Record Offsets Versus REXX Variable Substrings
This is probably the area that has the greatest potential to cause confusion:
So, to convert offsets to positions you subtract 3.
There are a couple of approaches:
In my code I chose the former - without the benefit of a conversion routine.
Extracting The SMFID
Because the SMFID is at offset 14, and bearing in mind the “subtract 3” point, you can extract it from a record in a variable with:
It’s a 4-byte character string - so it needs no further processing.
A section is a portion of the record with a fixed layout - and hence a fixed length. There might be more than one of a given layout.
Sections within a record are pointed to by data structures called triplets. As the name suggests, they consist of three fields:
To get to the sections of a given type you use the offset and then process them linearly, using the length to extract them, and to skip to the next.
In the SMF record header is a vector of triplets. So, for example, in SMF 70 Subtype 1 the vector starts at offset 28. The first few triplets in the vector are:
When I process a section I extract all three portions of the triplet separately:
/* Extract CPU Control Section position, length and count */ ccs=
Reminder: The control section starts at position 33 in the variable - which is offset 36.
Processing A Section
I extract the first (and only) CPU Control Section itself:
I’ve used the offset and the length in this substring operation.
Here offset 0 in the section is at position 1:
/* Extract Plant Of Manufacture */ pom=
In the above the “pom” field is at offset 74 for 4 in the CPU Control Section. I use
The Machine Serial Number is a bit more complex to extract:
In principle it’s a 16-byte character string, starting at offset 78 in the CPU Control Section. In practice it’s only the last 5 characters we want. Hence the second
Handling numeric fields is a bit trickier. When extracting the triplet fields you will’ve noticed the use of the
Timestamps come in a wide variety of formats, but let’s just concentrate on the SMF Timestamp - in SMF Date And Time (SMFDT) format.
Extract the date portion of the SMF timestamp with:
This is a little difficult to explain but I’ll give it a go:
The result will be a string like “2 Jun 2016”. I like to uppercase the month with:
parse upper value date2 with day month year
So much for the date. Let’s now do the time.
I’ve thrown away the seconds and hundredths of seconds but they’re not difficult to capture.
Input / Output
In terms of output formats you could create a CSV file, just by the appropriate syntactical sugaring. Similarly, using careful reformatting you could create a flat file that contains the numeric fields in binary format, etc.
While I wouldn’t recommend this approach for filtering all the records your systems cuts, you can do very sophisticated filtering. But I’ll stick to record types and subtypes here.
Record type is a single byte at offset 5, so you want an
Record subtype is generally4 two bytes at offset 22:
You can readily process SMF with REXX, starting with z/OS 2.1. I wouldn’t be keen to do it with high volume records - but most records cut by RMF are low volume. Exceptions are mostly SMF 74-1 (Disk/Tape Activity) and 74-5/8 (Disk Controller & Cache).
But for prototyping it’s quite a good match.