How to manage build dependencies

You can use DBB to track and manage build dependencies for full builds and incremental builds.

What does "build dependencies" mean

There are two types of build dependencies: direct and indirect build dependencies.

Direct build dependencies

z/OS programs often have references to other source files that are needed by the compiler to build the program. Examples of such references are the COBOL COPY, the PL/I %INCLUDE, the C #include, SQL INCLUDE, and the Assembler macro reference. These references are known as direct build dependencies and are important to identify the source code because:

  1. Precompilers and compilers need the latest version of the dependency source files to build the program correctly. This might require copying the dependency files from z/OS file system (zFS) directories to partition data sets (PDS) before building the program.
  2. They are needed to support building applications incrementally where only programs or their build dependencies that have changed since the last build are rebuilt.

Indirect build dependencies

Indirect build dependencies define relationships between the program source and the outputs from previous build processes. While indirect dependencies are not considered during dependency resolution, they are a factor in impact analysis.

Impact analysis refers to the process of identifying which programs in an application are impacted by code changes to copybooks or include files recursively. If a source file is changed and rebuilt, impact analysis can determine other sources that need to be rebuilt as a result of that output changing. An incremental build uses impact analysis to build only programs that are out of date either because the program has been modified or a copybook or include file that it uses has been modified since the last time the application was built.

There are two types of indirect build dependencies:

Type Explanation
Generated copybook One of the outputs from processing a BMS map is a copybook. A program source includes the copybook, not the BMS map. So there is an indirect build dependency between the program source and the BMS map.
Link dependency When program A statically links to program B, there is no direct link between these programs source files. However, when the source of program B is changed and rebuilt, impact analysis needs to know that program A must also be relinked.

How to handle build dependency relationships in DBB

You can use the two DBB scanners to discover the previously mentioned build dependency relationships: the source file scanner DependencyScanner and the link-edit scanner LinkEditScanner.

Type Scanner
Direct build dependencies DependencyScanner
Indirect build dependencies - Generated copybook DependencyScanner 1
Indirect build dependencies - Link dependency LinkEditScanner

Note: [1] You can use DependencyScanner to handle the generated copybook scenario by adding a dependency path to the BMS maps when defining resolution rules during impact analysis. For more information, see Indirect build dependencies in impact analysis.

You must address the following steps in your build scripts:

  1. Scanning source file dependencies
  2. Creating a collection to store source file dependency data
  3. Scanning static link dependencies
  4. Creating a collection to store link-edit dependencies
  5. Resolving logical build dependencies to physical files
  6. Identifying programs impacted by changed copybooks, include files and statically linked programs

1. Scanning source file dependencies

Dependency collection begins with scanning source code files for build dependencies. DBB provides a multi-language dependency scanner that can be used to find dependencies for Assembler, C/C++, COBOL, and PL/I source files. The DependencyScanner class is located in the com.ibm.dbb.dependency package.

// Scan a single file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def logicalFile = new DependencyScanner().scan(file, sourceDir)

// Scan an archived file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/copybook.tar.gz" // tar or tar.gz file
def logicalFileList = new DependencyScanner().scanArchive(file, sourceDir) // returns logical file list

// Scan files listed in an external file
def filelist = new File("/u/build/buildList.txt") as List<String>
def logicalFiles = [] as List<LogicalFile>
def scanner = new DependencyScanner()
filelist.each { file ->
   def logicalFile = scanner.scan(file, sourceDir)
   logicalFiles.add(logicalFile)
}

As seen in the example, the DependencyScanner scan method takes two String arguments:

The result of running the scan method is a LogicalFile that contains the dependency information of the scanned file. For more information about the LogicalFile class, see Resolving logical build dependencies to physical files.

The result of running the scanArchive method is a LogicalFile list that contains the dependency information of all of the files contained within the archive.

Language Hint

When the dependency scanner scans the source file, it automatically determines which programming language of the source file. On rare occasions, the scanner might misidentify a file's programming language. This can happen if the source file has little content or if the content is ambiguous. If this happens, the scanner will be unable to correctly identify the dependencies in the file. You can give the scanner a hint for the correct language by using the scanner.languageHint file property. When the file is scanned, the DependencyScanner checks to see whether the file is associated with a language hint property and will use it to determine the file's language. There are two ways to create a file property:

  1. In a properties file that is loaded via the BuildProperties.load(...) method 

    # Create a language hint file property for cobol copybooks
scanner.languageHint = COB :: **/copybook/*.cpy

  2. Using the BuildProperties static class in the build script

    // Create a language hint file property for cobol copybooks
BuildProperties.setFileProperty("scanner.languageHint", "COB", "**/copybook/*.cpy")

    Valid values for the language hint are:

Value Language
ASM Assembler
C C
CPP C++
COB COBOL
PLI PL/I

Handling code pages

The code page of the file is automatically determined in the following order:

  1. The file encoding tag of the source file is used if present.
    • Rocket's Git client automatically adds file encoding tags when cloning or pulling source files from a distributed Git server.
  2. The ZLANG environment variable is used if set.
  3. The default IBM-1047 code page is used.

Additionally the DependencyScanner.scan(file, sourceDir, encoding) allows the user to manually set code page of the source file.

2. Creating a collection to store source file dependency data

In order to use the build dependency information collected by the DependencyScanner for dependency resolution and impact analysis, all scanned source files (both programs and dependency files) will need their resulting logical files stored in the DBB repository database as part of a dependency Collection. A collection is a repository container for logical files. The scope of a collection is determined by the user. For example, a collection can contain all the logical files of a Git branch, from multiple Git repositories, or with other files added. This way the logical files in a collection can use the scanned file's relative path from the sourceDir as a unique identifier in the collection. Collections themselves can have any name but it is recommended to use the name of the Git branch of the source files being scanned.

Image of a LogicalFile
Example of a logical file stored in a DBB Repository Collection

Since collections are repository artifacts, communication with an active DBB server is required to create and modify them. For more information about initializing the RepositoryClient utility class, see Repository client.

// Create a collection to store scanned dependency data
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists("MortgageApplication.master"))
   client.createCollection("MortgageApplication.master")

Collection names must be unique in the DBB repository database, and an error will occur when you try to create a collection that already exists. A good practice is to first check if the collection with that name already exists before you create it.

Unlike the build result, a dependency collection is a simple repository object containing just a list of logical files. As such there is no dedicated interface for it. All collection APIs are located in the RepositoryClient utility class.

// Create a collection to store scanned dependency data
def collectionName = "MortgageApplication.master"
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists(collectionName))
   client.createCollection(collectionName) 

// Add or update a logical file to a collection
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def logicalFile = new DependencyScanner().scan(file, sourceDir)
client.saveLogicalFile(collectionName, logicalFile)

// Retrieve a logical file from a collection
logicalFile = client.getLogicalFile(collectionName, "MortgageApplication/cobol/epsnbrvl.cbl")

// Delete a logical file from a collection
client.deleteLogicalFile(collectionName, "MortgageApplication/cobol/epsnbrvl.cbl")

// Add a list of logical files to a collection
def filelist = new File("/u/build/buildList.txt") as List<String>
def logicalFiles = [] as List<LogicalFile>
def scanner = new DependencyScanner()
filelist.each { file ->
   def logicalFile = scanner.scan(file, sourceDir)
   logicalFiles.add(logicalFile)
}
client.saveLogicalFiles(collectionName, logicalFiles)

The RepositoryClient also provides two methods for searching a collection for logical dependencies:

// Find all logical files that have the logical name (lname) = EPSNBRVL i.e. MortgageApplication/cobol/epsnbrvl.cbl
def lfiles = client.getAllLogicalFiles(collectionName, "EPSNBRVL")

// Find all logical files that have a copy dependency on EPSMTCOM i.e. contain a COPY EPSMTCOM statement
def lname = "EPSMTCOM"
def category = "COPY"
def library = null  // null fields are ignored in the logical file search
def logicalDependency = new LogicalDependency(lname, library, category)
lfiles = client.getAllLogicalFiles(collectionName, logicalDependency)

// Find all logical files that have a dependency reference that resides in DD library MYLIB
logicalDependency = new LogicalDependency(null, "MYLIB", null)
lfiles = client.getAllLogicalFiles(collectionName, logicalDependency)

Link dependency collection begins with program object scanning. DBB provides a LinkEditScanner that scans program objects to determine the relationship between build process outputs and source files. This relationship is captured with link dependencies.

Note: Link-edit scanning is limited to program objects in a partitioned data set extended (PDSE). While the link-edit scanning works against a load module in a PDS, the information needed to perform impact analysis and dependency resolution isn't available. If the final result of a link-edit must be in a PDS, you are recommended to link-edit twice, once to a PDSE and again to a PDS, to obtain the necessary LINK dependencies from the PDSE. The PDSE can be discarded at the end of the build.

The LinkEditScanner class is located in the com.ibm.dbb.dependency package.

// Scan a single program object
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def logicalFile = new LinkEditScanner().scan(file, loadPDS, member)

As seen in the example, the LinkEditScanner scan method takes three String arguments:

The result of running the scan method is a LogicalFile that contains the link dependency information of the scanned program object. The LogicalFile class is covered more in depth in the Resolving logical build dependencies to physical files section.

The link-edit scanner returns the names of all dependencies that it finds in the program object. This might include dependencies related to other products (that is, CICS or Db2) or other program objects or object decks that are not managed by this build process. For each dependency that is found in the DBB repository, extra processing is required during impact analysis to process those dependencies.

To eliminate the extra processing, one can set an exclude filter on the LinkEditScanner to exclude dependencies from a particular data set, group of data sets, or individual members.

The filter contains a comma-separated list of patterns where an asterisk(*) is a wildcard and the last dotted segment is the module name. A filter of *.SUB1, *.SUB2 will exclude modules SUB1 and SUB2 from any data set. To exclude member HELLO in data set TEST.COBOL, use the pattern TEST.COBOL.HELLO. The pattern TEST.COBOL.* will match any member in the data set TEST.COBOL.

Any dependency that matches a pattern in the filter will be excluded from the set of dependencies returned from the scanner.

// Scan a single program object
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def scanner = new LinkEditScanner()
scanner.setExcludeFilter("DFH.V3R2M0.CICS.SDFHLOAD.*, CEE.SCEELKED.*");
def logicalFile = scanner.scan(file, loadPDS, member)

Alternatively, one can define a global property with the exclude filter so there is no need to set it for each instance of the scanner.

dbb.LinkEditScanner.excludeFilter = DFH.V3R2M0.CICS.SDFHLOAD.*, CEE.SCEELKED.*

In order for impact analysis to use the indirect dependency information collected by the link-edit scanner, all scanned program objects need their resulting logical files stored in the DBB repository database in a link dependency collection that is separate from the source file dependency data collection. A link dependency collection is a repository container for logical files that relate to the outputs from the link-edit scanner. Collections themselves can have any name, but it is recommended to use a similar but different name as the related dependency collection.

Note: The link dependency collection should be a separate collection from the source file dependency data collection. If placed into the same collection, the logical files and link dependencies may be overwritten when the source is rescanned. Both collections must be added to the ImpactResolver during impact analysis.

Because collections are repository artifacts, communication with an active DBB server is required to create and modify them. For more information about initializing the RepositoryClient utility class, see Repository client.

// Create a collection to store indirect dependency data
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists("MortgageApplication.master.outputs"))
   client.createCollection("MortgageApplication.master.outputs")

Collection names must be unique in the DBB repository database, and an error occurs when you try to create a collection that already exists. A good practice it to first check if the collection with that name already exists before creating it.

// Create a collection to store indirect dependency data
def collectionName = "MortgageApplication.master.outputs"
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists(collectionName))
   client.createCollection(collectionName) 

// Add or update a logical file to an indirect dependency collection
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def logicalFile = new LinkEditScanner().scan(file, loadPDS, member)
client.saveLogicalFile(collectionName, logicalFile)

5. Resolving logical build dependencies to physical files

As mentioned earlier, one of the primary reasons to collect and store application dependency data is to make sure that the build uses the latest versions of the build dependencies when compiling. However, all DBB has collected so far are the logical dependencies of a program. To find those dependencies, so that DBB can copy the latest versions to data sets for inclusion during compilation, you need to resolve their locations in the local file system.

You can use the DependencyResolver class to resolve the logical dependencies in a program's logical file to physical files in the local file system. It matches each dependency against a list of resolution rules provided by the user. A resolution rule is represented by the ResolutionRule class and consists of a match criteria (lname, category, library) and a search path in the collection of where the dependency file is located.

// Create a dependency resolver to resolve dependencies for program epscmort.cbl 
def resolver = new DependencyResolver()
revolver.setRepositoryClient(client)
resolver.setCollection("MortgageApplication.master")
resolver.setFile("MortgageApplication/cobol_cics_db2/epscmort.cbl")

// Create and add a resolution rule that matches dependencies whose library (DD) = SYSLIB
def rule = new ResolutionRule()
rule.setLibrary("SYSLIB")
resolver.addResolutionRule(rule)

// Create a search path for the MortgageApplication.master collection and the MortgageAppication/copybook directory
def path = new DependencyPath()
path.setCollection("MortgageApplication.master")
path.setSourceDir("/u/build/repo")
path.setDirectory("MortgageApplication/copybook")
rule.addPath(path)

// Create a search path for the MortgageApplication.master collection and the MortgageAppication/copybook directory using an archive
def archive_path = new DependencyPath()
archive_path.setCollection("MortgageApplication.master")
archive_path.setSourceDir("/u/build/repo") // source directory to the archive file
archive_path.setArchive("/MortgageApplication/copybook.tar.gz") // the archive file
archive_path.setDirectory("copybook/") // directory inside of the archive file to look (e.g. copybook/example.cpy)
rule.addPath(archive_path)

// Resolve the dependencies
def physicalDependencies = resolver.resolve()

The DependencyResolver class also supports setter method chaining.

def collectionName = "MortgageApplication.master"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def path = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/copybook")
def rule = new ResolutionRule().library("SYSLIB").path(path)
def resolver = new DependencyResolver().repositoryClient(client).collection(collectionName)file(file).rule(new ResolutionRule()
def physicalDependencies = resolver.resolve()

The result from running the DependencyResolver.resolve method is a list of physical dependencies, List<PhysicalDependency>. The PhysicalDependency class extends the LogicalDependency class with information needed to locate the dependency file in the local file system. The list of physical dependencies also includes indirect references. For example, if PGM1 references CPYBOOKA, and CPYBOOKA references CPYBOOKB, both CPYBOOKA and CPYBOOKB are in the list of physical dependencies for PGM1.

Notes:

  1. When applying a resolution rule, the DependencyResolver can run in two modes depends on whether the collection name exists:

    • If the collection name is present, the logical file is searched and retrieved from the webApp collection.

    • If the collection name is absent, the dependency resolver will scan the file in the file system immediately and create a logical file.

      You can add or omit the collection name in the following two places:

    • On the dependency resolver itself. This is used to get the initial logical file for the program that the resolver runs against.

    • In the Dependency Path(s) in the dependency rule. If the collection name is used in a dependency path, the search of the webApp collection for that path is used. If the dependency path omits the collection name and just uses the sourceDir and directory, the search is done on the file system with file scanning performed against files in the file system to produce logical files.

  2. Some dependencies like "DFHAID" found in programs that execute CICS commands or "SQLCA" found in programs that execute SQL queries are returned as unresolved dependencies. The reason is that they are not usually located in a user's SCM and therefore are not scanned and added to a DBB collection. However, this is not an issue since the reason for collecting the dependency information is to copy the files from zFS to data sets before the build. Since these dependencies already reside in data sets, nothing needs to be copied.

Now that the program's logical dependencies have been resolved to physical dependencies, they can be copied from their locations zFS to data sets for inclusion in compilation.

// Iterate through a list of physical depndencies to copy them to a data set
def physicalDependencies = resolver.resolve()
physicalDependencies.each { physDep ->
    if (physDep.isResolved()) {
       File file = new File("${physDep.getSourceDir()}/${physDep.getFile()}")
       new CopyToPDS().file(file).dataset("USR1.BUILD.COPYBOOK").member(physDep.getLname).execute()
    }
}

However, since the CopyToPDS command class has built-in support for copying a list of physical dependencies, the above example can be much simplified.

def physicalDependencies = resolver.resolve()
new CopyToPDS().dependencies(physicalDependencies).dataset("USR1.BUILD.COPYBOOK").execute()

6. Identifying programs impacted by changed copybooks, include files and statically linked programs

An incremental build uses impact analysis to only build programs that are out of date. For a more detailed example of creating an incremental build process using DBB, see the MortgageApplication/build/impacts.groovy build script, which is part of the DBB samples.

DBB provides the ImpactResolver class to perform impact analysis on changed copybooks and include files. Whereas the DependencyResolver starts with a program and finds all of the copybooks and include files that the program needs to build, the ImpactResolver goes the other way by starting with a copybook or include file and finds all of the programs and/or copybooks and include files that reference it. It does this by querying the collections in the repository to see what logical files have a dependency reference to the copybook or include file being searched.

All of the searches are run against DBB repository collections so communication with an active DBB server is required. For more information about initializing the RepositoryClient utility class, see Repository client. Also note that the more up-to-date the logical files in the collections to be searched, the more accurate the impact analysis is.

// Create an impact resolver to find programs and copybooks referencing epsmtinp.cpy
def resolver = new ImpactResolver()
revolver.setRepositoryClient(client)
resolver.addCollection("MortgageApplication.master")
resolver.setFile("MortgageApplication/copybook/epsmtinp.cpy")

// Create and add a resolution rule that matches dependencies whose library (DD) = SYSLIB
def rule = new ResolutionRule()
rule.setLibrary("SYSLIB")
resolver.addResolutionRule(rule)

// Create a search path for the MortgageApplication.master collection and the MortgageAppication/copybook directory
def path = new DependencyPath()
path.setCollection("MortgageApplication.master")
path.setSourceDir("/u/build/repo")
path.setDirectory("MortgageApplication/copybook")
rule.addPath(path)

def impactedFiles = resolver.resolve()

You might notice that setting up a ImpactResolver is very similar to setting up a DependencyResolver. This is because the impact resolver's initial query returns programs that potentially are impacted by changes to the copybook or include file being analyzed. It is only by applying the same resolution rules as used in dependency resolution can a determination be made that a program is impacted by the specific changed copybook or include file being analyzed.

The ImpactResolver class supports setter method chaining as well.

def collectionName = "MortgageApplication.master"
def file = "MortgageApplication/copybook/epsmtinp.cpy"
def path = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/copybook")
def rule = new ResolutionRule().library("SYSLIB").path(path)
def resolver = new ImpactResolver().repositoryClient(client).collection(collectionName).file(file).rule(new ResolutionRule()
def impactedFiles = resolver.resolve()

The result of running the ImpactResolver resolve method is a list of impact files, that is, List<ImpactFile>. An ImpactFile contains the following important information:

The list of impact files include both programs and intermediate copybooks and include files. For example, if a CPYBOOKB is referenced by CPYBOOKA, which itself is referenced by PRG1, then both CPYBOOKA and PGM1 will be in the list of impact files for CPYBOOKB.

Indirect build dependencies in impact analysis

As mentioned at the beginning of this topic, there are cases of indirect dependencies, like generated copybook from BMS maps, that can be handled by adding a path to the resolution rule. Since BMS copybooks are generated by the build process and programs that use them include the copybooks, the program dependencies are on a copybook. To resolve the dependency during impact analysis, the dependency on the BMS copybook name must resolve to the BMS map. In order to accomplish that, the directory containing the BMS map must be included in the resolution rule.

def cpypath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/copybook")
def bmspath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/bms")
def rule = new ResolutionRule().library("SYSLIB").path(cpypath).path(bmspath)

Other indirect dependencies, such as link dependencies, have additional requirements to resolve impacts. If program A statically links to program B, we need to know that program A must be rebuilt if program B is rebuilt. By using the LinkEditScanner scanner, you can gather this relationship and store that information in a link dependency collection. To resolve impacts using this information, you must include the link dependency collection when resolving impacts.

def collectionName = "MortgageApplication.master"
def outputsCollectionName = "MortgageApplication.master.outputs"
...
def resolver = new ImpactResolver().repositoryClient(client).collection(collectionName).collection(outputsCollectionName).file(file).rule(rule)
def impactedFiles = resolver.resolve()

Impact analysis must also be able to resolve a link dependency to a physical file. So, in order to resolve a link dependency on a program, the program's source directory must be added to a dependency path and a rule is created to resolve the link dependency, category LINK, to a physical file.

def collectionName = "MortgageApplication.master"
def outputsCollectionName = "MortgageApplication.master.outputs"
def cpypath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/copybook")
def bmspath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/bms")
def rule = new ResolutionRule().library("SYSLIB").path(cpypath).path(bmspath)
def cblpath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/cobol")
def lnkrule = new ResolutionRule().category("LINK").path(cblpath)
def resolver = new ImpactResolver().repositoryClient(client).collection(collectionName).collection(outputsCollectionName).file(file).rule(rule).rule(lnkrule)
def impactedFiles = resolver.resolve()

By using the cblpath and lnkrule above, in the case that program A has a link dependency on program B and program B is being rebuilt because its source changed or one of its dependencies changed, the impact resolver will be able to know that program A must also be rebuilt. The impact resolver will find the link dependency in the outputs collection and will be able to resolve that dependency to the source for program B using the resolution rule.

It is likely that the file structure of the application is more complicated than the MortgageApplication sample. Source might be broken up into more directories and sub-directories. In this case, additional dependency paths will be needed to cover link dependencies.