Tokenization

This section discusses some of the questions and answers regarding tokenization.

What does tokenization mean for z/OS Change Tracker?

A token represents the content of a PDS member or the content of a PS or DA file. For a PDS or PDSE one token is generated for each member.

Note: Tokenization is not supported for encrypted data sets.

How do I compare pairs of data sets when all data sets are available on the same processor?

There is no limit to the number of pairs that can be compared using z/OS Change Tracker. Data sets in the pair can be cataloged or uncataloged.

//CYGIN DD *
 OPTION=DIFF (MISMATCH, SAME, MODS ARE THE OTHER OPTIONS)
*
 NEWDSN=SYS1.PARMLIB
 OLDDSN=SYS2.PARMLIB
 COMPARE
*
 NEWDSN=SYS1.LINKLIB
 OLDDSN=SYS2.LINKLIB
 COMPARE
//*

Refer to the CYGT* sample jobs in Sample jobs.

How do I compare two groups of data sets when the data set names do not directly match?

Use the STRINGREPLACE (STRREPL) command. In this example, you build two lists of data sets in memory. The first list refers to a group of cataloged data sets starting with SYS1, and the second list refers to another group of data sets starting with SYS7. Using the STRREPL command, the lists are properly matched in memory. Once two data sets are matched by name, the member-by-member compare operation begins. This time, member names are matched and their tokens are evaluated to decide whether they have the same contents.

//CYGIN DD *
 OPTION=DIFF (MISMATCH, SAME, MODS ARE THE OTHER OPTIONS)
*
 STRREPL=('SYS1','SYS7')
*
 EXPLICIT=SYS1.PARMLIB
 EXPLICIT=SYS1.PROCLIB
 EXPLICIT=SYS1.VTAMLIB
 SHOW=LIST
*
 EXPLICIT=SYS7.PARMLIB
 EXPLICIT=SYS7.PROCLIB
 EXPLICIT=SYS7.VTAMLIB
 SHOW=LIST
*
 GROUPCOMPARE
//*

How do I compare two groups of data sets when one group is cataloged and the other group is not cataloged?

This example builds two lists of data sets and then uses the GROUPCOMPARE command to properly match the data set names. The first LIST refers to a group of cataloged data sets and the second LIST refers to a group of uncataloged data sets. After the two lists are created in memory, the data sets are matched. Once two data sets are matched, compare of the two data sets begin by matching the member names and evaluating their tokens to decide whether they have the same contents.

//CYGIN DD *
 OPTION=MODS (MISMATCH, SAME, DIFF ARE THE OTHER OPTIONS)
*
 EXPLICIT=SYS1.PARMLIB
 EXPLICIT=SYS1.PROCLIB
 EXPLICIT=SYS1.VTAMLIB
 SHOW=LIST
*
 INCLUDE=(VOL=RES001,DSN=SYS1.PARMLIB)
 INCLUDE=(VOL=RES001,DSN=SYS1.PROCLIB)
 INCLUDE=(VOL=RES001,DSN=SYS1.VTAMLIB)
 SHOW=LIST
 *
 GROUPCOMPARE
//*

Refer to the CYGT* sample jobs in Sample jobs.

Can I compare all data sets on two DASD volumes?

Yes. You can build two lists and then compare the data sets on both lists. Using the INCLUDE commands, data sets on two DASD volumes are placed in the lists in memory. Then, the GROUPCOMPARE is used to match the data set names and compare them.

//CYGIN DD *
 OPTION=DIFF (MISMATCH, SAME, MODS ARE THE OTHER OPTIONS)
*
 INCLUDE=(VOL=RES001,DSN=*)
 EXCLUDE=SYS1.BRODCAST
 EXCLUDE=SYS1.DUMP*
 SHOW=LIST
*
 INCLUDE=(VOL=RES002,DSN=*)
 EXCLUDE=SYS1.BRODCAST
 EXCLUDE=SYS1.DUMP*
 SHOW=LIST
*
GROUPCOMPARE
//*

How do I tokenize a group of explicit data sets?

The following statements tokenize a group of data sets and then store the tokens in the Control File. Storing the tokens in the Control File (the default method) provides an effective solution to identifying and reporting changes introduced to an environment over time.

//CYGIN DD *
 LOGCOMMENT='tokenize a list of explicit data sets - CICS GROUP'
*
 EXPLICIT=CICS.PROCLIB
 EXPLICIT=CICS.PARMLIB
 EXPLICIT=CICS.LOADLIB
*
 SHOW=LIST
 TOKEN=LIST
//*

Refer to the sample job CYGB1TOK in Sample jobs.

How do I tokenize uncataloged data sets?

The following statements show the use of multiple TOKEN commands. For uncataloged data sets, the VOL parameter must be specified. For cataloged data sets, the VOL parameter must be omitted.

//CYGIN DD *
 LOGCOMMENT='TOKENIZE MULTIPLE DATA SETS'
*
 TOKEN=(DSN=SYS1.PARMLIB,VOL=RES001)
 TOKEN=(DSN=SYS1.PARMLIB,VOL=RES002)
 TOKEN=(DSN=SYS1.PROCLIB)
 TOKEN=(DSN=SYS1.LINKLIB)
 TOKEN=(DSN=SYS1.VTAMLIB)
//*

Refer to the sample job CYGB1TOK in Sample jobs.

Can I tokenize patterns of data sets?

Yes. The following statements tokenize multiple patterns of data sets and then store the tokens in the Control File. Unwanted data set patterns can be excluded from the LIST using EXCLUDE commands.

//CYGIN DD *
 LOGCOMMENT='TOKENIZE A GROUP OF DATA SETS; MVS TEAM'
*
 PATTERN=SYS1.*
 PATTERN=SYS2.*
 EXCLUDE=SYS1.DUMP*
 EXCLUDE=SYS1.BRODCAST
*
 SHOW=LIST
 TOKEN=LIST
//*

Refer to the sample job CYGB1TOK in Sample jobs.

How do I identify and report changes in a list of previously tokenized data sets?

The following statements identify the changes in 3 data sets that were previously tokenized. This example assumes that the tokens have been stored in the Control File.

//CYGIN DD *
 OPTION=MODS
*
 EXPLICIT=CICS.PROCLIB
 EXPLICIT=CICS.PARMLIB
 EXPLICIT=CICS.LOADLIB
*
 SHOW=LIST
 TOKEN=(LIST,MODS)
//*

Refer to the sample job CYGB2MOD in Sample jobs.

Can I tokenize a group of libraries and store the tokens into an external file?

Yes. The following statements tokenize multiple data sets and then store the tokens to an external token file. By default, tokens are stored in the Control File unless the NOWRITE parameter in the TOKEN command is specified.

... 
//SYSUT2 DD DISP=SHR,DSN=node.OLDTOKEN
//CYGIN DD *
*
* A LIST OF EXPLICIT OR DATA SET PATTERNS TO BE TOKENIZED
*
 EXPLICIT=DEV.ATM.INSTALL
 EXPLICIT=DEV.ATM.LINKLIB
 EXPLICIT=DEV.ATM.SAMPLES
 SHOW=LIST
*
 TOKEN=(LIST,NOWRITE)
 CLOSE
//*

In this case, the tokens are stored in a PS file referenced by //SYSUT2 DD. This method is recommended if the tokens need to be transferred to a remote environment as a representation of the first environment.

The attributes of the token file should be:

DCB=(DSORG=PS,RECFM=FB,LRECL=4096,BLKSIZE=4096)

Refer to the sample job CYGD1TOK in Sample jobs.

How do I compare two groups of software not available to the same processor?

As shown in the previous example, two environments can be tokenized and the external token files can be brought together on the same processor. Token files can be transferred between systems using protocols such as NDM, XMIT, or FTP, or they can be sent as e-mail attachments.

The following statements show how two token files can be compared to identify and report the differences in the configuration and the contents.

... 
//DD1 DD DISP=SHR,DSN=node.NEWTOKEN <== NEW TOKEN FILE
//DD2 DD DISP=SHR,DSN=node.OLDTOKEN <== OLD TOKEN FILE
//CYGIN DD *
 OPTION=MODS
*
 DD1SHOW
 SHOW=LIST BUILDS THE LIST1 IN MEMORY
*
 DD2SHOW
 SHOW=LIST BUILDS THE LIST2 IN MEMORY
*
 REMOTEGROUPCOMP
//*

Refer to the sample job CYGD3CMP in Sample jobs.

Can I synchronize two groups of local libraries?

Yes. The following statements refer to multiple pairs of libraries. Each pair is internally tokenized and then the tokens are compared in memory to create a list of members being different in their contents (the DIFF members). Then, using the COPYDIFF command these members are copied from the source PDS to the target PDS. Hence, if there are only a few members different only those members are copied rather than copying the entire PDS. Moreover, the members existing in the target PDS but missing from the source PDS are optionally deleted (DELETEMEM=YES) from the target to achieve identical pairs of PDS. Only the target PDS is the subject of change and the source PDS remains unaltered.

The TESTRUN=YES command provides a simulation run showing the scope of change between the two environments without an actually copying or deleting any members.

//CYGIN DD *
TESTRUN=NO
COMPRESS=YES
*
DELETEMEM=YES
DSN1=IBMU11.TEST.PDS1
DSN2=IBMU12.TEST.PDS1
COPYDIFF
*
DELETEMEM=NO
DSN1=IBMU11.TEST.PDSE2
DSN2=IBMU12.TEST.PDSE2
COPYDIFF
//*

Refer to the sample job CYGB5CPY in Sample jobs.

Can I reconcile differences between pairs of uncataloged data sets?

Yes. The first pair of partitioned data sets in the sample job below is uncataloged. The VOL1 and VOL2 commands must refer to the volumes where these data sets are located on. In the second pair, only the first data set in the pair is uncataloged and the second one is a cataloged data set.

//CYGIN DD *
TESTRUN=NO
COMPRESS=YES
*
VOL1=VOLSER
VOL2=VOLSER
DSN1=IBMU11.TEST.PDS1
DSN2=IBMU12.TEST.PDS1
COPYDIFF
*
VOL1=VOLSER
DSN1=IBMU11.TEST.PDSE2
DSN2=IBMU12.TEST.PDSE2 /* ONLY DSN2 IS CATALOGED */
COPYDIFF
//*

Refer to the sample job CYGB6CPY in Sample jobs.

Can I use z/OS Change Tracker to synchronize our production software with the Disaster Recovery software?

Yes. To learn more about this capability refer to the CYGD* jobs in Sample jobs.

How do I verify whether two software environments are identical in their configuration and their contents?

Using the z/OS Change Tracker tokenization capability, this task is quite simple. By tokenizing each environment and storing the tokens on external token files, a representation of each environment is established. Then, these token files must be made available on the same processor for comparison by the REMOTEGROUPCOMP command.

Refer to the CYGT* jobs in Sample jobs.