How to organize the build script

The helloworld.groovy build script that we write in the tutorial is a simple example of how to use Groovy to build z/OS programs by using DBB APIs. However, it has two problems to be fixed before it can be used for building an enterprise application:

• Problem 1: Everything is hardcoded in the script. The program name, the source, the working directories, and the referenced data sets are all defined in the script. Hardcoding not only prevents the build script from building any program other than helloworld.cbl, but also potentially ties the build script to a specific build machine where those data sets and directories exist. And only developers and builders who have access to those data sets and directories would be able to run it.

• Problem 2: The script can only build COBOL programs. Most z/OS applications are composed of different types programming languages, such as COBOL, PL/I, Assembler, C, and BMS Maps. Altough it is possible to create a single build script that builds any kind of program that it might encounter, the script would be large and contain rather complex conditional logic.

Solutions to Problem 1: Use properties, properties files, YAML files, or all

Properties are name-value pairs of type string that are defined outside of the build scripts and are either loaded (read) from a file or passed in as command line arguments. Below is a sample properties file:

# Absolute path of the source directory i.e. directory containing .git
sourceDir=/u/app1/src

# Absolute path to the build output directory
workDir=/u/app1/build/work

# High Level Qualifier for build data sets
hlq=APP1.BUILD

# DBB Repository Web Application authentication properties
dbb.RepositoryClient.url=https://localhost:9443/dbb
dbb.RepositoryClient.userId=ADMIN
dbb.RepositoryClient.passwordFile=${sourceDir}/MortgageApplication/build/ADMIN.pw

# DBB Repository Dependency Data Collection Name
collection=MortgageApplication

You can also use YAML files, which are of file extension .yml or .yaml, to configure the build just as using properties files. You can use one or more YAML files or a combination of YAML and properties files. For more information, see Using YAML files to configure builds. Below is a sample YAML file that defines some of the properties in the properties file above.

---
buildProperties:

  # DBB Repository Web Application authentication properties
  - key: dbb.RepositoryClient.url
    value:
  - key: dbb.RepositoryClient.userId
    value: ADMIN
  - key: dbb.RepositoryClient.passwordFile
    value: ${sourceDir}/Build/MortgageApplication/build/ADMIN.pw

# DBB Repository Dependency Data Collection Name
  - key: collection
    value: MortgageApplication

Though most scripting languages support passing in string arguments from the command line, depending on the number of properties needed by the build scripts, it might be easier to define the properties in a properties file. However, it is also a common practice to define a properties file as the default properties but also write the build script to allow users to override the default properties with passed-in arguments, as discussed in Passing script arguments from command line.

Categorization of properties

The properties are categorized as follows:

• Sandbox properties
• Build properties class
  • File properties
• DBB configuration properties

Sandbox properties

In the hello world example, the helloworld.cbl file was located in the /u/usr1/build directory. You can tell from the path that the directory probably belongs to a specific user 'usr1'.

• If another user wanted to use the helloworld.groovy build script, they must modify it to use their directory instead.
• In addition to /u/usr1/build being the source directory for the build script, it is also where the build script is copying the SYSPRINT log file to. In general, the work directory is where non-binary outputs of the build process, such as logs, temporary files, and build metadata, are created.
• The two data sets that are being written to by the build script have the same high-level qualifier (HLQ) USR1.BUILD.

These three locations comprise what is commonly called the "user sandbox". It is where the user can store modified source code and generate build outputs without overwriting anyone else files or data set members. The sandbox properties are good candidates to be included a properties file. The location of a user's properties file could be passed into the build script as a single argument. A common convention is to have a file called build.properties in the same directory as the build script that could contain a default sandbox to be used to team builds. The content of which could be:

sourceDir=/u/app1/src
workDir=/u/app1/build/work
hlq=APP1.BUILD

Build properties class

Properties and properties files are fairly common and most scripting languages have native support for them. Groovy, as a Java scripting language, uses the java.util.Properties class to read properties files. This class is perfectly adequate to support what has been discussed so far; however, DBB does provide the com.ibm.dbb.build.BuildProperties class as well. The BuildProperties class has much of the same functionality as the Java Properties class but it also provides three additional useful features:

• There is a single static BuildProperties instance. Therefore, the BuildProperties is always available to all scripts without being passed from one script to another.

• BuildProperties values can reference other build properties. Property references are evaluated dynamically at run time.

• It introduces a new concept called file properties. File properties are properties that are associated to a set of files or even a single file.

File properties

File properties are useful in cases where some source files being built need to be handled differently than the rest of the source files. An example of this is where some COBOL programs need different compiler options than the rest of the COBOL programs. Instead of hardcoding the names of the special case files in the build scripts, file properties can be created for those source files in the build.properties file. This allows for easy maintenance if you need to add or remove files of this type.

compileOpts=LIB
compileOpts=APOST,DATA31,LIB,DYNAM::App1/cobol/*.cbl,App2/cobol/*.cbl

In the example above, the first property defined is a normal property called compileOpts that is used as the 'default' compile option for the COBOL compiler. The second property defined is a file property. This can be seen by the use of :: as a delimiter between the property value and the comma-delimited list of file path patterns use to indicate which files the property is associated to. In this case, all of the files in App1/cobol and App2/cobol directories with the .cbl file extension are associated with compileOpts=APOST,DATA31,LIB,DYNAM. All other programs will default to compileOpts=LIB.

Note: Normally when two properties are defined with the same property name, the second declaration of the property will overwrite the first declaration of the property. This rule also applies to the BuildProperties class but only applies for normal key-value pair properties. File properties allow for duplicate property names because the 'key' of a file property is name + file pattern. If two file properties have both the name and a file path pattern that is identical, then the value of the second declaration of the name + file pattern will overwrite the first one.

To use the file property in the build script you first will have to load the build.properties file using a BuildProperties.load(...) method. Then you can use the BuildProperties class name as a static variable in our script:

def opts = BuildProperties.getFileProperty("compileOpts", file)
def compile = new MVSExec().pgm("IGYCRCTL").parm(opts)

In the case where file = App1/cobol/ABXDTC.cbl, then APOST,DATA31,LIB,DYNAM will be used as the parm value. If the file = App3/cobol/DEBWRD.cbl, then LIB will be used as the parm value.

DBB configuration properties

Some DBB APIs require you to set configuration like properties on each declaration that are often the same for all declarations and never change during the build process. Examples of this can be seen in the RepositoryClient, ISPFExec, TSOExec and so on. You can declare DBB configuration properties in the properties files that will automatically set those values without having to code them explicitly in the build scripts.

Below is a list of supported DBB configuration properties:

Using YAML files to configure builds

Each YAML file must have a top-level element buildProperties followed by multiple key and value pairs. Other top-level elements can exist but are ignored.

Passing script arguments from command line

Another way to customize build scripts is to pass arguments to the script from the command line when invoking the script or in the case of a CI server like Jenkins, there is usually a field in build script plug-in where input arguments are listed. This does not have to be an either/or scenario. A common practice is to have the build script use a properties file as its default properties but also write the build script to allow users to override the default properties with passed in arguments. This is another useful way in handling sandbox properties by having the team build sandbox properties defined in the build.properties file and then allow users to pass in their own sandbox properties for when they want to run a user build.

CliBuilder

Groovy has a utility class called CliBuilder that is very useful for automatically parsing command line arguments which contain both optional and required arguments. It can even output a usage message if desired. For example:

def parseInput(String[] cliArgs){
    def cli = new CliBuilder(usage: "deploy.groovy [options]")
    cli.b(longOpt:'buztool', args:1, argName:'file', 'Absolute path to UrbanCode Deploy buztool.sh script') 
    cli.w(longOpt:'workDir', args:1, argName:'dir', 'Absolute path to the DBB build output directory')
    cli.c(longOpt:'component', args:1, argName:'name', 'Name of the UCD component to create version in')
    cli.h(longOpt:'help', 'Prints this message')
    def opts = cli.parse(cliArgs)
    if (opts.h) { // if help option used, print usage and exit
         cli.usage()
        System.exit(0)
    }
}

Solutions to Problem 2: Use a main build script to call more dedicated build scripts

Most enterprise applications are comprised of many source files often written in a variety of programming languages, such as COBOL, PL/I, C and Assembler. They may also include specialized source files such as CICS BMS and IMS MFS map files as well as database DBRM files and Link-Edit files. Putting all of the logic to build an application this complex into a single build script would result in a very large and complicated script. A better design would be to have several build scripts with each designed to build a single type of source file. The build process would begin by calling a "main" build script that could perform a number of build initialization tasks such as the following:

• Handling command line arguments
• Loading properties files
• Creating directories and data sets
• Initializing build artifacts
• Scanning files for new dependency data

After the build initialization is complete, the main build script iterates though a build file list, calling each dedicated build script depending on the file type to build. After all the files in the build list have been processed, the main build script could run through a clean up phase, deleting temporary files and finalizing and storing build artifacts.

Build file list

The Mortgage Application sample, which is part of the samples that ships with DBB, contains a build script build.groovy in it's build folder. This is the main build script for the Mortgage Application. It is expecting a file to be passed in as an argument. This is the name of the file to build. However if the file has a .txt file extension, then the build script assumes that it is a text file which contains a list of files that need to be built.

Note: In this documentation and the DBB API descriptions, the term 'file' can have two different meanings depending on context.

• In some cases, it refers to a physical file on the build machine that has an absolute path i.e. /u/usr1/src/MortgageApplication/build/build.properties. As when referring to log, properties, text files, or even a source file, that is the source of a CopyToPDS statement. In the DBB API documentation, these usage of 'file' is usually typed as a java.io.File.
• In other instances as in this case, 'file' refers to a source file that is to be built or scanned. It is the unique ID of the file in the repository. If the repository is Git, it is the relative path of the file. When loaded from a Git server onto the local build machine, it is the relative path from the source directory MortgageApplication/cobol/epsnbrvl.cbl. When used in this context in the DBB API documentation, the 'file' is usually typed as a String.

The build list file can be a static file that contains a list of files that are always to be built. Alternatively, it can be dynamically created by a pre-build process to contain only the files that need to be built at that time. An example of a static build list file is included in the MortgageApplication sample in MortgageApplication/build/files.txt:

MortgageApplication/bms/epsmlis.bms
MortgageApplication/bms/epsmort.bms
MortgageApplication/cobol/epsmlist.cbl
MortgageApplication/cobol/epsmpmt.cbl
MortgageApplication/cobol/epsnbrvl.cbl
MortgageApplication/cobol_cics/epscsmrd.cbl
MortgageApplication/cobol_cics/epscsmrt.cbl
MortgageApplication/cobol_cics_db2/epscmort.cbl
MortgageApplication/link/epsmlist.lnk
MortgageApplication/mfs/dfsiv1.mfs
MortgageApplication/copybook/epsmortf.cpy
MortgageApplication/copybook/epsmtcom.cpy
MortgageApplication/copybook/epsmtinp.cpy
MortgageApplication/copybook/epsmtout.cpy
MortgageApplication/copybook/epsnbrpm.cpy
MortgageApplication/copybook/epspdata.cpy

Since this file is used for both dependency scanning and build programs, the list contains both programs and copybooks. However, only the programs and map files will be built. This is done by using script mappings.

Script mappings

Script mappings are file properties in which the property name is scriptMapping and the value is the name of the specific build script that should be called to build the files associated to the property. Script mapping supports specification of criteria such as file extensions, file paths, glob and regex patterns, for example:

scriptMapping = BMSProcessing :: MortgageApplication/bms/*.bms
scriptMapping = Compile :: MortgageApplication/cobol/epsmlist.cbl, MortgageApplication/cobol/epsnbrvl.cbl
scriptMapping = CobolCompile :: MortgageApplication/cobol/epsmpmt.cbl, MortgageApplication/cobol_cics/*.cbl, MortgageApplication/cobol_cics_db2/*.cbl
scriptMapping = LinkEdit :: MortgageApplication/link/*.lnk
scriptMapping = MFSGENUtility :: MortgageApplication/mfs/*.mfs

These properties are loaded at the beginning of the build process with the other build properties. You can see the referred called build scripts in the MortgageApplication/build folder:

BMSProcessing.groovy
CobolCompile.groovy
Compile.groovy
LinkEdit.groovy
MFSGENUtility.groovy

Though script mappings can be accessed as file properties using the DBB BuildProperties class, DBB also provides a special utility class called com.ibm.dbb.build.ScriptMappings that contains additional methods for processing script mappings. It contains useful methods such as isMapped(file), getScriptName(file) and getMappedList(scriptName, buildList) which returns a sublist of all the files in the build list that are mapped to a specific script. An example of using ScriptMappings.getMappedList(scriptName, buildList) to get a list of files mapped to specific script and the iterate over the list calling the build script for each file can be seen in MortgageApplication/build/build.groovy:

// Use the ScriptMappings class to get the files mapped to the build script
def scriptList = ScriptMappings.getMappedList(script, buildList)
def scriptName = "$properties.sourceDir/MortgageApplication/build/${script}.groovy"
scriptList.each { file ->
    run(new File(scriptName), [file] as String[])
    processCounter++
}

Reducing time consumption by Groovy build script caching

During build execution, the called Groovy scripts are compiled immediately into Java classes and invoked. Though the compilation is fast, it still takes time and if the build process builds hundreds or even thousands of source files, this can result in significant build processing times. Since most build processes generally call the same few Groovy scripts over and over again, adding a Groovy script caching mechanism to your build process can greatly reduce the amount of time it takes to load, recompile and execute Groovy scripts resulting in lower process times. DBB provides a Groovy base script class com.ibm.dbb.groovy.ScriptLoader that you can extend when writing your build scripts that will automatically load and cache called Groovy scripts.

Script caching API

You can extend your build scripts by adding the following line to the beginning of the script:

@groovy.transform.BaseScript com.ibm.dbb.groovy.ScriptLoader baseScript

This now provides the following built-in methods that the build script can now invoke:

• void runScript(File script, String[] args=[]) - This method is used to call another executable Groovy script.
  • Will automatically try to use the cached version of the compiled script class or updates the cache if the script is being called for the first time during the build process.
  • Can be used to call both DBB ScriptLoader based Groovy scripts and generic Groovy scripts
• void runScript(File script, Map<String,Object> argMap) - This method is similar to the previous method but passes an argument Map instead of a String array.
  • For use only when calling another DBB ScriptLoader based Groovy script
    • Allows for the easy passing of objects between scripts
    • A Map property called argMap is created in the target script
  • If the target script is not a DBB ScriptLoader based Groovy script, then an empty String [] args is passed instead.
• GroovyObject loadScript(File script) - This method performs automatic caching similar to the other methods described but only loads the script without trying to run it.
  • This method can be used to load object-oriented Groovy scripts, that is, Tools.groovy
• getScriptDir() - Returns the String representation of the parent directory of the running script.

Note: The File script argument in the above methods supports both absolute and relative paths. If the script parameter contains a relative path, then it is automatically appended to the current script's parent directory. This means that loadScript(new File("Tools.groovy")) will try to load the Tools.groovy script file in the same directory that the current running script is in.

Example:

@groovy.transform.BaseScript com.ibm.dbb.groovy.ScriptLoader baseScript
import com.ibm.dbb.build.*

// receive passed arguments from argument map
def file = argMap.file
. . .

// call the bind package script using script caching API and passing argument hashmap
runScript(new File("BindPackage.groovy"), ["file":file])