Migrating data sets to Git
Even though Git is not a prerequisite of DBB, the existence of a Git client on z/OS was the driving factor for the creation of DBB. Most DBB users are assumed to install the Rocket Software port of the Git client to z/OS as part of their DevOps solution. To facilitate the move to Git as an SCM for z/OS development, you can use a DBB migration tool that copies source code from partitioned data sets (PDS) to a local Git repository on the z/OS File System (zFS) to be committed and pushed to a distributed Git server.
Setting up a local Git repository on zFS
If you have not installed Rocket Git client on your USS system, follow the instructions in the Setting up Git on USS.
Create a local Git repository on zFS as follows:
- Follow your company's policy to create an empty repository in your Git server
- Create or navigate to a zFS directory as the local Git repository for your source code
- Clone the empty repository to your USS system
git clone git@USERNAME/AppName.git (or what you have named it)
Migrating by using DBB migration tool
After the local Git repository has been created on zFS, you can start using the the migration tool to copy source files from data set libraries to the local repository. In addition to copying the source from PDS to zFS, the migration tool also creates
and updates the .gitattributes
file. The .gitattributes
file is required by Rocket's Git client to perform automatic codepage conversion between the Git server and the local Git repository during the migration.
Run the migration tool by executing a shell script called migrate.sh
which is located in /usr/lpp/IBM/dbb/migration/bin
. The tool has two modes of operation:
- Run migration using a mapping file
- Run migration using a mapping rule
Running migration using a mapping file
The easiest way to use the migration tool is to provide it a mapping file to process.
migrate.sh -r /var/localGit/appname /u/migrate/mappingFile.txt
In the above example, the -r /var/localGit/appname
option refers to the local Git repository where the .gitattributes
file will be generated and /u/migrate/mappingFile.txt
is the location of the mapping file.
Each line in the mapping file maps a fully qualified PDS member to the absolute path of the zFS file it is to be copied. Each line can have an optional PDS encoding value if the encoding of the PDS member is not the default IBM-1047
.
# Begin a line with '#' for comments which are ignored by the migration tool
APPNAME.COBOL(ABC) /var/localGit/appname/cobol/abc.cbl
APPNAME.COPYBOOK(XYZ) /var/localGit/appname/copybook/xyz.cpy pdsEncoding=IBM-037
The challenge with this method is providing a mapping file to process. You can use any method to generate the mapping file of the forementioned format. One of the options is to run the migration tool using a mapping rule (see below) to generate a mapping file created by the mapping rule without actually performing the migration. You can examine the generated mapping file to verify the result and then execute the migration tool using the generated mapping file.
Running migration using a mapping rule
The second mode of operation for the migration tool is to define a mapping rule to be applied to the members of one or more data sets. Internally, this results in a list of mappings similar to what is found in a mapping file. You can choose to perform the migration using this internal list or to generate the list as a mapping file and terminate without performing the actual migration.
Here is the usage of the migration tool:
usage: migrate.sh [options] <data sets>
Use this migration tool to migrate members from data sets to a local GIT
repository on HFS.
Options:
-m,--mapping <mapping> The mapping rule (optional), for
example:
-m MappingRule[extension:cbl,toLower:true]
-o,--output <output> Output of the generated mapping file
(optional)
-p,--preview Perform a dry-run to generate a mapping
file (optional)
-r,--repository <repository> Local GIT repository to migrate to
(required)
The format of the mapping rule option is:
-m MappingRule[attrName:value,attrName:value]
The mapping rule supports the following attributes:
hlq
- the HLQ of the data sets being migrated.extension
- append a file extension to each member.toLower
- convert the whole zFS path including file name to lower case.pdsEncoding
- specify the encoding to read the contents of all members in the list datasets.pdsMapping
- indicate whether the data set structure should be used to convert to sub-directories in the repository directory. Example:ONLINE.COBOL(ABC) -> /var/localGit/appname/online/cobol/abc.cbl
. Default is true. If false, then all members will be copied to repository directory or targetDir if provided.targetDir
- the sub-directory of the local GIT repository to copy the members to.tagFile
- (Optional) a file contains a list of members and their tag.tagMapping
- (Optional) whether to add tags to the directory path. The default value istrue
.
Examples
In the examples below, you can skip the actual migration and just generate a mapping file by adding the following options to the migrate.sh
command:
-p -o <new/mapping/file/location.txt>
Migrate all members in data sets USER.COBOL
and USER.BMS
to a local GIT repository at /u/user/repo
using the default mapping rule attributes, which converts each segment in data set name to a folder, for example,
copying USER.COBOL(HELLO)
to /u/user/repo/USER/COBOL/HELLO
:
migrate.sh -r /u/user/repo USER.COBOL,USER.BMS
Migrate all members in data set USER.COBOL
to /u/user/repo
and convert the directories and files to lower case and add .cbl extension to the files, for example: copying USER.COBOL(HELLO)
to /u/user/repo/user/cobol/hello.cbl
:
migrate.sh -r /u/user/repo -m MappingRule[extension:CBL,toLower:true] USER.COBOL
Migrate all members in data set USER.COBOL
and USER.COPYBOOK
to /u/user/repo using HLQ
:
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CBL,toLower:true,pdsEncoding:IBM-1037] COBOL
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CPY,toLower:true,pdsEncoding:IBM-1037] COPYBOOK
Migrate members whose name start with BLZ in USER.COBOL
data set to directory /u/user/repo/COBOL
:
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CBL] COBOL(BLZ*)
Migrate members whose name start with BLZ in USER.COBOL
to directory /u/user/repo/myApp/COBOL
using attribute 'targetDir':
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CBL,targetDir:myApp] COBOL(BLZ*)
Migrate members whose name start with BLZ in USER.COBOL
to directory /u/user/repo/myApp
using attributes 'targetDir' and 'pdsMapping':
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,extension:CBL,targetDir:myApp,pdsMapping:false] COBOL(BLZ*)
Generate a mapping file for migrating all members from data set USER.COBOL
to /u/user/repo
. The result then can be used for subsequent migration command; see the next sample. Notice: no members are being copied during this
process.
migrate.sh -p -r /u/user/repo -m MappingRule[extension:CBL,toLower:true] -o /u/user/mappings.txt USER.COBOL
Migrate using a generated mapping file. See above example on how to generate a mapping file.
migrate.sh -r /u/user/repo /u/user/mappings.txt
Assume that you have 2 sets of data sets that contain COBOL and COPYBOOKS, for example, USER.COBOL
and USER.COPYBOOK
where USER.COBOL
contains members like BLUEXXX, REDXXX, YELLOWXX. You would copy them to
/u/user/repo
such that members BLUEXXX of COBOL and COPYBOOK are going to /u/user/repo/blue/cobol
and /u/user/repo/blue/copybook
respectively. Members REDXXX are going to /u/user/repo/red/cobol
and /u/user/repo/red/copybook
. The rest (that are not start with BLUE or RED) are going to /u/user/repo/others/cobol
and .u/user/repo/others/copybook
.
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,targetDir:blue,toLower:true] COBOL(BLUE*),COPYBOOK(BLUE*)
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,targetDir:red,toLower:true] COBOL(RED*),COPYBOOK(RED*)
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,targetDir:others,toLower:true] COBOL(regex:^(?!BLUE|RED)),COPYBOOK(regex:^(?!BLUE|RED))
Assume you have a tag file with the following content (note that the members and tags are separated by a space):
USER.COBOL(PGM1) COBE
USER.COBOL(PGM2) COBCICD2
USER.COBOL(PGM3) COBE
You run the following command:
migrate.sh -r /u/user/repo -m MappingRule[hlq:USER,toLower:true] COBE
The PGM1
and PGM3
members are copied:
USER.COBOL(PGM1)
is copied to /u/user/repo/cobol/cobe/pgm1
.
USER.COBOL(PGM3)
is copied to /u/user/repo/cobol/cobe/pgm3
.
The PGM2
member is not copied since it is not tagged with COBE
.
Managing non-roundtrippable characters
The migration tool does not do any encoding conversion when migrating members to HFS, so if members are encoded in EBCDIC, the files copied to HFS are also encoded in EBCDIC. However, source files that are stored in distributed Git repositories are usually encoded in UTF-8. When round-trip conversion is mentioned in this documentation, it refers to the following process:
- Convert character set from EBCDIC to UTF-8 when it is committed to Git, and
- Convert it back from UTF-8 to EBCDIC when it is loaded to HFS from the Git repository
There are some situations where this round-trip conversion does not preserve the original content of the source files. This is often referred to as non-roundtrippable character situations. For example:
-
New line character in the record
The record can contain new line separators (0x15=NL; 0D=CR; 0x25=LF). When such files are committed to Git, these new line characters are converted to a brand new line. Therefore, after the round trip conversion, the loaded members would have additional empty record compare to the original migrated member.
-
Empty Shift-Out and Shift-In
The Shift-Out (0x0E) and Shift-In (0x0F) are often used to embed different character sets in a text file. For example, embed a double-byte character sets in a source file. When such files are committed to Git, these characters are stripped off and the double-byte character sets (DBCS) are converted. When these files are loaded from Git to HFS, the Shift-Out and Shift-In characters are re-inserted. In the case of an empty Shift-Out and Shift-In (0E0F), these characters are stripped off but are never re-inserted later. Therefore, the content of loaded member would not be exactly the same as the original migrated member.
The migration tool detects such characters and handles the corresponding files as follows:
- When you run the migration tool in preview mode without a mapping file, the tool mark these files as
BINARY
in the.gitattributes
file and the mapping file that is generated when you specify-o
. - When you run the migration tool with the mapping file, the tool copies these files using binary mode, making them transferable between Git and z/OS.
Finishing migration process
When all of the source files have been copied to the local Git repository, you are ready to push the files to the Git server as follows:
-
Add the newly copied files to the local Git repository database by navigating to the local Git repository directory and perform a Git 'add'.
git add .
-
Commit the changes specifying an appropriate commit message.
git commit -m "Migrate source libraries to Git"
-
Push the changes to the central git server.
git push