Migrating data sets to Git

Even though Git is not a prerequisite of DBB, the existence of a Git client on z/OS was the driving factor for the creation of DBB. Most DBB users are assumed to install the Rocket Software port of the Git client to z/OS as part of their DevOps solution. To facilitate the move to Git as an SCM for z/OS development, you can use a DBB migration tool that copies source code from partitioned data sets (PDS) to a local Git repository on the z/OS File System (zFS) to be committed and pushed to a distributed Git server.

Setting up a local Git repository on zFS

If you have not installed Rocket Git client on your USS system, follow the instructions in the Setting up Git on USS.

Create a local Git repository on zFS as follows:

Follow your company's policy to create an empty repository in your Git server
Create or navigate to a zFS directory as the local Git repository for your source code
Clone the empty repository to your USS system

git clone git@USERNAME/AppName.git (or what you have named it)

Migrating by using DBB migration tool

After the local Git repository has been created on zFS, you can start using the the migration tool to copy source files from data set libraries to the local repository. In addition to copying the source from PDS to zFS, the migration tool also creates and updates the .gitattributes file. The .gitattributes file is required by Rocket's Git client to perform automatic codepage conversion between the Git server and the local Git repository during the migration.

Run the migration tool by executing a shell script called migrate.sh which is located in /usr/lpp/IBM/dbb/migration/bin. The tool has two modes of operation:

Run migration using a mapping file
Run migration using a mapping rule

Running migration using a mapping file

The easiest way to use the migration tool is to provide it a mapping file to process.

migrate.sh -r /var/localGit/appname /u/migrate/mappingFile.txt

In the above example, the -r /var/localGit/appname option refers to the local Git repository where the .gitattributes file will be generated and /u/migrate/mappingFile.txt is the location of the mapping file.

Each line in the mapping file maps a fully qualified PDS member to the absolute path of the zFS file it is to be copied. Each line can have an optional PDS encoding value if the encoding of the PDS member is not the default IBM-1047.

# Begin a line with '#' for comments which are ignored by the migration tool
APPNAME.COBOL(ABC) /var/localGit/appname/cobol/abc.cbl
APPNAME.COPYBOOK(XYZ) /var/localGit/appname/copybook/xyz.cpy pdsEncoding=IBM-037

The challenge with this method is providing a mapping file to process. You can use any method to generate the mapping file of the forementioned format. One of the options is to run the migration tool using a mapping rule (see below) to generate a mapping file created by the mapping rule without actually performing the migration. You can examine the generated mapping file to verify the result and then execute the migration tool using the generated mapping file.

Running migration using a mapping rule

The second mode of operation for the migration tool is to define a mapping rule to be applied to the members of one or more data sets. Internally, this results in a list of mappings similar to what is found in a mapping file. You can choose to perform the migration using this internal list or to generate the list as a mapping file and terminate without performing the actual migration.

Here is the usage of the migration tool:

usage: migrate.sh [options] <data sets>
Use this migration tool to migrate members from data sets to a local GIT
repository on HFS.
Options:
 -m,--mapping <mapping>         The mapping rule (optional), for
                                example:
                                -m MappingRule[extension:cbl,toLower:true]
 -o,--output <output>           Output of the generated mapping file
                                (optional)
 -p,--preview                   Perform a dry-run to generate a mapping
                                file (optional)
 -r,--repository <repository>   Local GIT repository to migrate to
                                (required)

The format of the mapping rule option is:

-m MappingRule[attrName:value,attrName:value]

The mapping rule supports the following attributes:

hlq - the HLQ of the data sets being migrated.
extension - append a file extension to each member.
toLower - convert the whole zFS path including file name to lower case.
pdsEncoding - specify the encoding to read the contents of all members in the list datasets.
pdsMapping - indicate whether the data set structure should be used to convert to sub-directories in the repository directory. Example: ONLINE.COBOL(ABC) -> /var/localGit/appname/online/cobol/abc.cbl. Default is true. If false, then all members will be copied to repository directory or targetDir if provided.
targetDir - the sub-directory of the local GIT repository to copy the members to.
tagFile - (Optional) a file contains a list of members and their tag.
tagMapping - (Optional) whether to add tags to the directory path. The default value is true.

Examples

In the examples below, you can skip the actual migration and just generate a mapping file by adding the following options to the migrate.sh command:

-p -o <new/mapping/file/location.txt>

Migrate all members in data sets USER.COBOL and USER.BMS to a local GIT repository at /u/user/repo using the default mapping rule attributes, which converts each segment in data set name to a folder, for example, copying USER.COBOL(HELLO) to /u/user/repo/USER/COBOL/HELLO:

migrate.sh -r /u/user/repo USER.COBOL,USER.BMS

Migrate all members in data set USER.COBOL to /u/user/repo and convert the directories and files to lower case and add .cbl extension to the files, for example: copying USER.COBOL(HELLO) to /u/user/repo/user/cobol/hello.cbl:

migrate.sh -r /u/user/repo -m MappingRule[extension:CBL,toLower:true] USER.COBOL

Migrate all members in data set USER.COBOL and USER.COPYBOOK to /u/user/repo using HLQ:

migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CBL,toLower:true,pdsEncoding:IBM-1037] COBOL

migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CPY,toLower:true,pdsEncoding:IBM-1037] COPYBOOK

Migrate members whose name start with BLZ in USER.COBOL data set to directory /u/user/repo/COBOL:

migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CBL] COBOL(BLZ*)

Migrate members whose name start with BLZ in USER.COBOL to directory /u/user/repo/myApp/COBOL using attribute 'targetDir':

migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CBL,targetDir:myApp] COBOL(BLZ*)

Migrate members whose name start with BLZ in USER.COBOL to directory /u/user/repo/myApp using attributes 'targetDir' and 'pdsMapping':

migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CBL,targetDir:myApp,pdsMapping:false] COBOL(BLZ*)

Generate a mapping file for migrating all members from data set USER.COBOL to /u/user/repo. The result then can be used for subsequent migration command; see the next sample. Notice: no members are being copied during this process.

migrate.sh -p -r /u/user/repo -m MappingRule[extension:CBL,toLower:true] -o /u/user/mappings.txt USER.COBOL

Migrate using a generated mapping file. See above example on how to generate a mapping file.

migrate.sh -r /u/user/repo /u/user/mappings.txt

Assume that you have 2 sets of data sets that contain COBOL and COPYBOOKS, for example, USER.COBOL and USER.COPYBOOK where USER.COBOL contains members like BLUEXXX, REDXXX, YELLOWXX. You would copy them to /u/user/repo such that members BLUEXXX of COBOL and COPYBOOK are going to /u/user/repo/blue/cobol and /u/user/repo/blue/copybook respectively. Members REDXXX are going to /u/user/repo/red/cobol and /u/user/repo/red/copybook. The rest (that are not start with BLUE or RED) are going to /u/user/repo/others/cobol and .u/user/repo/others/copybook.

migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,targetDir:blue,toLower:true] COBOL(BLUE*),COPYBOOK(BLUE*)
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,targetDir:red,toLower:true] COBOL(RED*),COPYBOOK(RED*)
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,targetDir:others,toLower:true] COBOL(regex:^(?!BLUE|RED)),COPYBOOK(regex:^(?!BLUE|RED))

Assume you have a tag file with the following content (note that the members and tags are separated by a space):

USER.COBOL(PGM1) COBE
USER.COBOL(PGM2) COBCICD2
USER.COBOL(PGM3) COBE

You run the following command:

migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,toLower:true] COBE

The PGM1 and PGM3 members are copied:

USER.COBOL(PGM1) is copied to /u/user/repo/cobol/cobe/pgm1.

USER.COBOL(PGM3) is copied to /u/user/repo/cobol/cobe/pgm3.

The PGM2 member is not copied since it is not tagged with COBE.

Managing non-roundtrippable characters

The migration tool does not do any encoding conversion when migrating members to HFS, so if members are encoded in EBCDIC, the files copied to HFS are also encoded in EBCDIC. However, source files that are stored in distributed Git repositories are usually encoded in UTF-8. When round-trip conversion is mentioned in this documentation, it refers to the following process:

Convert character set from EBCDIC to UTF-8 when it is committed to Git, and
Convert it back from UTF-8 to EBCDIC when it is loaded to HFS from the Git repository

There are some situations where this round-trip conversion does not preserve the original content of the source files. This is often referred to as non-roundtrippable character situations. For example:

New line character in the record

The record can contain new line separators (0x15=NL; 0D=CR; 0x25=LF). When such files are committed to Git, these new line characters are converted to a brand new line. Therefore, after the round trip conversion, the loaded members would have additional empty record compare to the original migrated member.
Empty Shift-Out and Shift-In

The Shift-Out (0x0E) and Shift-In (0x0F) are often used to embed different character sets in a text file. For example, embed a double-byte character sets in a source file. When such files are committed to Git, these characters are stripped off and the double-byte character sets (DBCS) are converted. When these files are loaded from Git to HFS, the Shift-Out and Shift-In characters are re-inserted. In the case of an empty Shift-Out and Shift-In (0E0F), these characters are stripped off but are never re-inserted later. Therefore, the content of loaded member would not be exactly the same as the original migrated member.

The migration tool detects such characters and handles the corresponding files as follows:

When you run the migration tool in preview mode without a mapping file, the tool mark these files as BINARY in the .gitattributes file and the mapping file that is generated when you specify -o.
When you run the migration tool with the mapping file, the tool copies these files using binary mode, making them transferable between Git and z/OS.

Finishing migration process

When all of the source files have been copied to the local Git repository, you are ready to push the files to the Git server as follows:

Add the newly copied files to the local Git repository database by navigating to the local Git repository directory and perform a Git 'add'.
```
 git add .
```
Commit the changes specifying an appropriate commit message.
```
 git commit -m "Migrate source libraries to Git"
```
Push the changes to the central git server.
```
 git push
```