How to manage build dependencies
You can use DBB to track and manage build dependencies for full builds and incremental builds.
What does "build dependencies" mean
There are two types of build dependencies: direct and indirect build dependencies.
Direct build dependencies
z/OS programs often have references to other source files that are needed by the compiler to build the program. Examples of such references are the COBOL COPY
, the PL/I %INCLUDE
, the C #include
, SQL INCLUDE,
and the Assembler macro reference. These references are known as direct build dependencies and are important to identify the source code because:
- Precompilers and compilers need the latest version of the dependency source files to build the program correctly. This might require copying the dependency files from z/OS file system (zFS) directories to partition data sets (PDS) before building the program.
- They are needed to support building applications incrementally where only programs or their build dependencies that have changed since the last build are rebuilt.
Indirect build dependencies
Indirect build dependencies define relationships between the program source and the outputs from previous build processes. While indirect dependencies are not considered during dependency resolution, they are a factor in impact analysis.
Impact analysis refers to the process of identifying which programs in an application are impacted by code changes to copybooks or include files recursively. If a source file is changed and rebuilt, impact analysis can determine other sources that need to be rebuilt as a result of that output changing. An incremental build uses impact analysis to build only programs that are out of date either because the program has been modified or a copybook or include file that it uses has been modified since the last time the application was built.
There are two types of indirect build dependencies:
Type | Explanation |
---|---|
Generated copybook | One of the outputs from processing a BMS map is a copybook. A program source includes the copybook, not the BMS map. So there is an indirect build dependency between the program source and the BMS map. |
Link dependency | When program A statically links to program B, there is no direct link between these programs source files. However, when the source of program B is changed and rebuilt, impact analysis needs to know that program A must also be relinked. |
How to handle build dependency relationships in DBB
You can use the two DBB scanners to discover the previously mentioned build dependency relationships: the source file scanner DependencyScanner
and the link-edit scanner LinkEditScanner
.
Type | Scanner |
---|---|
Direct build dependencies | DependencyScanner |
Indirect build dependencies - Generated copybook | DependencyScanner 1 |
Indirect build dependencies - Link dependency | LinkEditScanner |
Note: [1] You can use DependencyScanner
to handle the generated copybook scenario by adding a dependency path to the BMS maps when defining resolution rules during impact analysis. For more information, see Indirect build dependencies in impact analysis.
You must address the following steps in your build scripts:
- Scanning source file dependencies
- Creating a collection to store source file dependency data
- Scanning static link dependencies
- Creating a collection to store link-edit dependencies
- Resolving logical build dependencies to physical files
- Identifying programs impacted by changed copybooks, include files and statically linked programs
1. Scanning source file dependencies
Dependency collection begins with scanning source code files for build dependencies. DBB provides a multi-language dependency scanner that can be used to find dependencies for Assembler, C/C++, COBOL, and PL/I source files. The DependencyScanner
class is located in the com.ibm.dbb.dependency
package.
// Scan a single file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def logicalFile = new DependencyScanner().scan(file, sourceDir)
// Scan an archived file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/copybook.tar.gz" // tar or tar.gz file
def logicalFileList = new DependencyScanner().scanArchive(file, sourceDir) // returns logical file list
// Scan files listed in an external file
def filelist = new File("/u/build/buildList.txt") as List<String>
def logicalFiles = [] as List<LogicalFile>
def scanner = new DependencyScanner()
filelist.each { file ->
def logicalFile = scanner.scan(file, sourceDir)
logicalFiles.add(logicalFile)
}
As seen in the example, the DependencyScanner scan method takes two String
arguments:
- file - This is the source file's path relative to the sourceDir of the file to scan. It is used as the logical file's key when it is stored in the DBB repository.
- sourceDir - This is the absolute path of the directory on the local file system that contains the source files. If the source directory is a local Git repository, it is the path of the directory that contains the .git folder. It is used to locate the file in the local file system in order to scan its contents.
The result of running the scan method is a LogicalFile
that contains the dependency information of the scanned file. For more information about the LogicalFile class, see Resolving logical build dependencies to physical files.
The result of running the scanArchive
method is a LogicalFile
list that contains the dependency information of all of the files contained within the archive.
Language Hint
When the dependency scanner scans the source file, it automatically determines which programming language of the source file. On rare occasions, the scanner might misidentify a file's programming language. This can happen if the source file
has little content or if the content is ambiguous. If this happens, the scanner will be unable to correctly identify the dependencies in the file. You can give the scanner a hint for the correct language by using the scanner.languageHint
file property. When the file is scanned, the DependencyScanner checks to see whether the file is associated with a language hint property and will use it to determine the file's language. There are two ways to create a file property:
-
In a properties file that is loaded via the
BuildProperties.load(...)
method# Create a language hint file property for cobol copybooks scanner.languageHint = COB :: **/copybook/*.cpy
-
Using the BuildProperties static class in the build script
// Create a language hint file property for cobol copybooks BuildProperties.setFileProperty("scanner.languageHint", "COB", "**/copybook/*.cpy")
Valid values for the language hint are:
Value | Language |
---|---|
ASM | Assembler |
C | C |
CPP | C++ |
COB | COBOL |
PLI | PL/I |
Handling code pages
The code page of the file is automatically determined in the following order:
- The file encoding tag of the source file is used if present.
- Rocket's Git client automatically adds file encoding tags when cloning or pulling source files from a distributed Git server.
- The ZLANG environment variable is used if set.
- The default IBM-1047 code page is used.
Additionally the DependencyScanner.scan(file, sourceDir, encoding)
allows the user to manually set code page of the source file.
2. Creating a collection to store source file dependency data
In order to use the build dependency information collected by the DependencyScanner
for dependency resolution and impact analysis, all scanned source files (both programs and dependency files) will need their resulting logical files
stored in the DBB repository database as part of a dependency Collection. A collection is a repository container for logical files. The scope of a collection is determined by the user. For example, a collection can contain all the logical
files of a Git branch, from multiple Git repositories, or with other files added. This way the logical files in a collection can use the scanned file's relative path from the sourceDir as a unique identifier in the collection. Collections
themselves can have any name but it is recommended to use the name of the Git branch of the source files being scanned.
Example of a logical file stored in a DBB Repository Collection
Since collections are repository artifacts, communication with an active DBB server is required to create and modify them. For more information about initializing the RepositoryClient utility class, see Repository client.
// Create a collection to store scanned dependency data
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists("MortgageApplication.master"))
client.createCollection("MortgageApplication.master")
Collection names must be unique in the DBB repository database, and an error will occur when you try to create a collection that already exists. A good practice is to first check if the collection with that name already exists before you create it.
Unlike the build result, a dependency collection is a simple repository object containing just a list of logical files. As such there is no dedicated interface for it. All collection APIs are located in the RepositoryClient utility class.
// Create a collection to store scanned dependency data
def collectionName = "MortgageApplication.master"
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists(collectionName))
client.createCollection(collectionName)
// Add or update a logical file to a collection
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def logicalFile = new DependencyScanner().scan(file, sourceDir)
client.saveLogicalFile(collectionName, logicalFile)
// Retrieve a logical file from a collection
logicalFile = client.getLogicalFile(collectionName, "MortgageApplication/cobol/epsnbrvl.cbl")
// Delete a logical file from a collection
client.deleteLogicalFile(collectionName, "MortgageApplication/cobol/epsnbrvl.cbl")
// Add a list of logical files to a collection
def filelist = new File("/u/build/buildList.txt") as List<String>
def logicalFiles = [] as List<LogicalFile>
def scanner = new DependencyScanner()
filelist.each { file ->
def logicalFile = scanner.scan(file, sourceDir)
logicalFiles.add(logicalFile)
}
client.saveLogicalFiles(collectionName, logicalFiles)
The RepositoryClient also provides two methods for searching a collection for logical dependencies:
- getAllLogicalFiles(collectionName, logicalName) - finds all the logical files in a collection that match the logical name being searched. Use this to include the dependency source files to build the program correctly.
- getAllLogicalFiles(collectionName, LogicalDependency) - finds all the logical files in a collection that match the set fields of a logical dependency. Use this for impact analysis during incremental builds.
// Find all logical files that have the logical name (lname) = EPSNBRVL i.e. MortgageApplication/cobol/epsnbrvl.cbl
def lfiles = client.getAllLogicalFiles(collectionName, "EPSNBRVL")
// Find all logical files that have a copy dependency on EPSMTCOM i.e. contain a COPY EPSMTCOM statement
def lname = "EPSMTCOM"
def category = "COPY"
def library = null // null fields are ignored in the logical file search
def logicalDependency = new LogicalDependency(lname, library, category)
lfiles = client.getAllLogicalFiles(collectionName, logicalDependency)
// Find all logical files that have a dependency reference that resides in DD library MYLIB
logicalDependency = new LogicalDependency(null, "MYLIB", null)
lfiles = client.getAllLogicalFiles(collectionName, logicalDependency)
3. Scanning static link dependencies
Link dependency collection begins with program object scanning. DBB provides a LinkEditScanner that scans program objects to determine the relationship between build process outputs and source files. This relationship is captured with link dependencies.
Note: Link-edit scanning is limited to program objects in a partitioned data set extended (PDSE). While the link-edit scanning works against a load module in a PDS, the information needed to perform impact analysis and dependency resolution isn't available. If the final result of a link-edit must be in a PDS, you are recommended to link-edit twice, once to a PDSE and again to a PDS, to obtain the necessary LINK dependencies from the PDSE. The PDSE can be discarded at the end of the build.
The LinkEditScanner class is located in the com.ibm.dbb.dependency
package.
// Scan a single program object
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def logicalFile = new LinkEditScanner().scan(file, loadPDS, member)
As seen in the example, the LinkEditScanner scan method takes three String
arguments:
- file - This is the source file's path relative to the sourceDir of the file to scan. It is used as the logical file's key when it is stored in the DBB repository.
- loadPDS - The fully qualified PDS containing the program object.
- member - The program object member name.
The result of running the scan method is a LogicalFile
that contains the link dependency information of the scanned program object. The LogicalFile
class is covered more in depth in the Resolving logical build dependencies to physical files section.
Link-Edit scanning exclude filter
The link-edit scanner returns the names of all dependencies that it finds in the program object. This might include dependencies related to other products (that is, CICS or Db2) or other program objects or object decks that are not managed by this build process. For each dependency that is found in the DBB repository, extra processing is required during impact analysis to process those dependencies.
To eliminate the extra processing, one can set an exclude filter on the LinkEditScanner to exclude dependencies from a particular data set, group of data sets, or individual members.
The filter contains a comma-separated list of patterns where an asterisk(*) is a wildcard and the last dotted segment is the module name. A filter of *.SUB1, *.SUB2 will exclude modules SUB1 and SUB2 from any data set. To exclude member HELLO in data set TEST.COBOL, use the pattern TEST.COBOL.HELLO. The pattern TEST.COBOL.* will match any member in the data set TEST.COBOL.
Any dependency that matches a pattern in the filter will be excluded from the set of dependencies returned from the scanner.
// Scan a single program object
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def scanner = new LinkEditScanner()
scanner.setExcludeFilter("DFH.V3R2M0.CICS.SDFHLOAD.*, CEE.SCEELKED.*");
def logicalFile = scanner.scan(file, loadPDS, member)
Alternatively, one can define a global property with the exclude filter so there is no need to set it for each instance of the scanner.
dbb.LinkEditScanner.excludeFilter = DFH.V3R2M0.CICS.SDFHLOAD.*, CEE.SCEELKED.*
4. Creating a collection to store link-edit dependencies
In order for impact analysis to use the indirect dependency information collected by the link-edit scanner, all scanned program objects need their resulting logical files stored in the DBB repository database in a link dependency collection that is separate from the source file dependency data collection. A link dependency collection is a repository container for logical files that relate to the outputs from the link-edit scanner. Collections themselves can have any name, but it is recommended to use a similar but different name as the related dependency collection.
Note: The link dependency collection should be a separate collection from the source file dependency data collection. If placed into the same collection, the logical files and link dependencies may be overwritten when the source is rescanned. Both collections must be added to the ImpactResolver during impact analysis.
Because collections are repository artifacts, communication with an active DBB server is required to create and modify them. For more information about initializing the RepositoryClient utility class, see Repository client.
// Create a collection to store indirect dependency data
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists("MortgageApplication.master.outputs"))
client.createCollection("MortgageApplication.master.outputs")
Collection names must be unique in the DBB repository database, and an error occurs when you try to create a collection that already exists. A good practice it to first check if the collection with that name already exists before creating it.
// Create a collection to store indirect dependency data
def collectionName = "MortgageApplication.master.outputs"
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists(collectionName))
client.createCollection(collectionName)
// Add or update a logical file to an indirect dependency collection
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def logicalFile = new LinkEditScanner().scan(file, loadPDS, member)
client.saveLogicalFile(collectionName, logicalFile)
5. Resolving logical build dependencies to physical files
As mentioned earlier, one of the primary reasons to collect and store application dependency data is to make sure that the build uses the latest versions of the build dependencies when compiling. However, all DBB has collected so far are the logical dependencies of a program. To find those dependencies, so that DBB can copy the latest versions to data sets for inclusion during compilation, you need to resolve their locations in the local file system.
You can use the DependencyResolver
class to resolve the logical dependencies in a program's logical file to physical files in the local file system. It matches each dependency against a list of resolution rules provided
by the user. A resolution rule is represented by the ResolutionRule
class and consists of a match criteria (lname, category, library) and a search path in the collection of where the dependency file is located.
// Create a dependency resolver to resolve dependencies for program epscmort.cbl
def resolver = new DependencyResolver()
revolver.setRepositoryClient(client)
resolver.setCollection("MortgageApplication.master")
resolver.setFile("MortgageApplication/cobol_cics_db2/epscmort.cbl")
// Create and add a resolution rule that matches dependencies whose library (DD) = SYSLIB
def rule = new ResolutionRule()
rule.setLibrary("SYSLIB")
resolver.addResolutionRule(rule)
// Create a search path for the MortgageApplication.master collection and the MortgageAppication/copybook directory
def path = new DependencyPath()
path.setCollection("MortgageApplication.master")
path.setSourceDir("/u/build/repo")
path.setDirectory("MortgageApplication/copybook")
rule.addPath(path)
// Create a search path for the MortgageApplication.master collection and the MortgageAppication/copybook directory using an archive
def archive_path = new DependencyPath()
archive_path.setCollection("MortgageApplication.master")
archive_path.setSourceDir("/u/build/repo") // source directory to the archive file
archive_path.setArchive("/MortgageApplication/copybook.tar.gz") // the archive file
archive_path.setDirectory("copybook/") // directory inside of the archive file to look (e.g. copybook/example.cpy)
rule.addPath(archive_path)
// Resolve the dependencies
def physicalDependencies = resolver.resolve()
The DependencyResolver
class also supports setter method chaining.
def collectionName = "MortgageApplication.master"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def path = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/copybook")
def rule = new ResolutionRule().library("SYSLIB").path(path)
def resolver = new DependencyResolver().repositoryClient(client).collection(collectionName)file(file).rule(new ResolutionRule()
def physicalDependencies = resolver.resolve()
The result from running the DependencyResolver.resolve
method is a list of physical dependencies, List<PhysicalDependency>
. The PhysicalDependency
class extends the LogicalDependency
class
with information needed to locate the dependency file in the local file system. The list of physical dependencies also includes indirect references. For example, if PGM1
references CPYBOOKA
, and CPYBOOKA
references CPYBOOKB
, both CPYBOOKA
and CPYBOOKB
are in the list of physical dependencies for PGM1
.
Notes:
When applying a resolution rule, the
DependencyResolver
can run in two modes depends on whether the collection name exists:
If the collection name is present, the logical file is searched and retrieved from the webApp collection.
If the collection name is absent, the dependency resolver will scan the file in the file system immediately and create a logical file.
You can add or omit the collection name in the following two places:
On the dependency resolver itself. This is used to get the initial logical file for the program that the resolver runs against.
In the Dependency Path(s) in the dependency rule. If the collection name is used in a dependency path, the search of the webApp collection for that path is used. If the dependency path omits the collection name and just uses the sourceDir and directory, the search is done on the file system with file scanning performed against files in the file system to produce logical files.
Some dependencies like "DFHAID" found in programs that execute CICS commands or "SQLCA" found in programs that execute SQL queries are returned as unresolved dependencies. The reason is that they are not usually located in a user's SCM and therefore are not scanned and added to a DBB collection. However, this is not an issue since the reason for collecting the dependency information is to copy the files from zFS to data sets before the build. Since these dependencies already reside in data sets, nothing needs to be copied.
Now that the program's logical dependencies have been resolved to physical dependencies, they can be copied from their locations zFS to data sets for inclusion in compilation.
// Iterate through a list of physical depndencies to copy them to a data set
def physicalDependencies = resolver.resolve()
physicalDependencies.each { physDep ->
if (physDep.isResolved()) {
File file = new File("${physDep.getSourceDir()}/${physDep.getFile()}")
new CopyToPDS().file(file).dataset("USR1.BUILD.COPYBOOK").member(physDep.getLname).execute()
}
}
However, since the CopyToPDS command class has built-in support for copying a list of physical dependencies, the above example can be much simplified.
def physicalDependencies = resolver.resolve()
new CopyToPDS().dependencies(physicalDependencies).dataset("USR1.BUILD.COPYBOOK").execute()
6. Identifying programs impacted by changed copybooks, include files and statically linked programs
An incremental build uses impact analysis to only build programs that are out of date. For a more detailed example of creating an incremental build process using DBB, see the MortgageApplication/build/impacts.groovy
build script, which
is part of the DBB samples.
DBB provides the ImpactResolver
class to perform impact analysis on changed copybooks and include files. Whereas the DependencyResolver
starts with a program and finds all of the copybooks and include files that the program
needs to build, the ImpactResolver
goes the other way by starting with a copybook or include file and finds all of the programs and/or copybooks and include files that reference it. It does this by querying the collections in the
repository to see what logical files have a dependency reference to the copybook or include file being searched.
All of the searches are run against DBB repository collections so communication with an active DBB server is required. For more information about initializing the RepositoryClient utility class, see Repository client. Also note that the more up-to-date the logical files in the collections to be searched, the more accurate the impact analysis is.
// Create an impact resolver to find programs and copybooks referencing epsmtinp.cpy
def resolver = new ImpactResolver()
revolver.setRepositoryClient(client)
resolver.addCollection("MortgageApplication.master")
resolver.setFile("MortgageApplication/copybook/epsmtinp.cpy")
// Create and add a resolution rule that matches dependencies whose library (DD) = SYSLIB
def rule = new ResolutionRule()
rule.setLibrary("SYSLIB")
resolver.addResolutionRule(rule)
// Create a search path for the MortgageApplication.master collection and the MortgageAppication/copybook directory
def path = new DependencyPath()
path.setCollection("MortgageApplication.master")
path.setSourceDir("/u/build/repo")
path.setDirectory("MortgageApplication/copybook")
rule.addPath(path)
def impactedFiles = resolver.resolve()
You might notice that setting up a ImpactResolver
is very similar to setting up a DependencyResolver
. This is because the impact resolver's initial query returns programs that potentially are impacted by changes
to the copybook or include file being analyzed. It is only by applying the same resolution rules as used in dependency resolution can a determination be made that a program is impacted by the specific changed copybook or include file being analyzed.
The ImpactResolver
class supports setter method chaining as well.
def collectionName = "MortgageApplication.master"
def file = "MortgageApplication/copybook/epsmtinp.cpy"
def path = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/copybook")
def rule = new ResolutionRule().library("SYSLIB").path(path)
def resolver = new ImpactResolver().repositoryClient(client).collection(collectionName).file(file).rule(new ResolutionRule()
def impactedFiles = resolver.resolve()
The result of running the ImpactResolver
resolve method is a list of impact files, that is, List<ImpactFile>
. An ImpactFile
contains the following important information:
- collection - The name of the collection the impacted program or copybook is located in.
- file - This is the program's or copybook's path relative to the sourceDir of the file to scan. It is the same value as the file attribute in a logical file.
The list of impact files include both programs and intermediate copybooks and include files. For example, if a CPYBOOKB
is referenced by CPYBOOKA
, which itself is referenced by PRG1
, then both CPYBOOKA
and PGM1 will be in the list of impact files for CPYBOOKB
.
Indirect build dependencies in impact analysis
As mentioned at the beginning of this topic, there are cases of indirect dependencies, like generated copybook from BMS maps, that can be handled by adding a path to the resolution rule. Since BMS copybooks are generated by the build process and programs that use them include the copybooks, the program dependencies are on a copybook. To resolve the dependency during impact analysis, the dependency on the BMS copybook name must resolve to the BMS map. In order to accomplish that, the directory containing the BMS map must be included in the resolution rule.
def cpypath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/copybook")
def bmspath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/bms")
def rule = new ResolutionRule().library("SYSLIB").path(cpypath).path(bmspath)
Other indirect dependencies, such as link dependencies, have additional requirements to resolve impacts. If program A statically links to program B, we need to know that program A must be rebuilt if program B is rebuilt. By using the LinkEditScanner
scanner, you can gather this relationship and store that information in a link dependency collection. To resolve impacts using this information, you must include the link dependency collection when resolving impacts.
def collectionName = "MortgageApplication.master"
def outputsCollectionName = "MortgageApplication.master.outputs"
...
def resolver = new ImpactResolver().repositoryClient(client).collection(collectionName).collection(outputsCollectionName).file(file).rule(rule)
def impactedFiles = resolver.resolve()
Impact analysis must also be able to resolve a link dependency to a physical file. So, in order to resolve a link dependency on a program, the program's source directory must be added to a dependency path and a rule is created to resolve the link dependency, category LINK, to a physical file.
def collectionName = "MortgageApplication.master"
def outputsCollectionName = "MortgageApplication.master.outputs"
def cpypath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/copybook")
def bmspath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/bms")
def rule = new ResolutionRule().library("SYSLIB").path(cpypath).path(bmspath)
def cblpath = new DependencyPath().collection(collectionName).sourceDir("/u/build/repo").directory("MortgageApplication/cobol")
def lnkrule = new ResolutionRule().category("LINK").path(cblpath)
def resolver = new ImpactResolver().repositoryClient(client).collection(collectionName).collection(outputsCollectionName).file(file).rule(rule).rule(lnkrule)
def impactedFiles = resolver.resolve()
By using the cblpath
and lnkrule
above, in the case that program A has a link dependency on program B and program B is being rebuilt because its source changed or one of its dependencies changed, the impact resolver will
be able to know that program A must also be rebuilt. The impact resolver will find the link dependency in the outputs collection and will be able to resolve that dependency to the source for program B using the resolution rule.
It is likely that the file structure of the application is more complicated than the MortgageApplication sample. Source might be broken up into more directories and sub-directories. In this case, additional dependency paths will be needed to cover link dependencies.