Using Bad File Finder to identify changes in behavior between builds

The Bad File Finder tool performs a binary search that determines if a single file introduces a change in behavior between two different, but compatible, builds of a program. It also identifies the file in question. This tool can be very useful when migrating or upgrading from one version of classic XL or Open XL to another or when making optimization level changes. The Bad File Finder tool is located in /opt/IBM®/openxlC/17.1.2/tools and is named bad_file_finder.sh. Access the help using the command bad_file_finder.sh [-h]. Examples can be found in /opt/IBM/openxlC/17.1.2/tools/bad_file_finder/examples.

When to use Bad File Finder

Note: The Bad File Finder tool is available in versions 17.1.2.5 and newer of the compiler.

Two typical scenarios where bad_file_finder.sh can be used are:

Migration or upgrade - moving to a different release of the compiler
Optimization changes - when changing optimization levels results in different behavior

Other situations where the tool may prove useful are:

When a change is made to one file that causes an observable effect in a different file.

For example, changing the size of a memory allocation may result in a crash in a different file at a point in the execution that is far removed from the cause of the failure. In such scenarios, the search method employed by bad_file_finder.sh would identify the file containing the change to the memory allocation as the culprit file, as opposed to the file containing the point of failure.
When an observable effect may be different from the file containing the change is one involving changes to files that are included in other files.

For example, template definitions, inline functions and constants are often specified in header files and can cause changes to code in multiple files. It may be known that a particular change to a header file is problematic but it may not be known which including file is affected adversely by the change. In such situations, bad_file_finder.sh would identify the file containing the code that is adversely affected by the change, rather than the header file.

The Bad File Finder tool will observe the build processes for a sample of each of the before and after builds (called "good" and "bad" builds, respectively, for ease of reference) and prepare proxies for the compilers. It will instruct the user to perform builds using these proxies. Then, after each build, it will wait for the user to indicate whether the build was "good" or "bad". The tool will use this information to perform a search for the minimal files necessary to observe the "bad" behavior, automatically mixing good and bad versions of the files in the build.

Important:

All commands issued to the tool pertaining to a particular search need to be issued in the same directory in which the start command was issued.

In situations where multiple files are necessary to show the change of behavior, the Bad File Finder tool will attempt to find a reduced set of files that exhibit the change of behavior. Programs are compatible when they are link compatible. In general, C programs compiled with IBM compilers will be compatible. For C++, different versions of the same series of compilers are likely compatible. However, xlC is not compatible with xlclang++ nor with ibm-clang++. Also, xlclang++ and ibm-clang++ are generally compatible, but programs compiled with IPA or LTO are likely not compatible. The Bad File Finder tool considers each compiled object file as distinct, so build systems that use the same object file more than once may produce inconsistent results.

Note: Distributed build systems are not handled.

How Bad File Finder works

The steps that comprise a build are controlled by the user. Therefore, building and testing a build can consist of an arbitrary number of steps. The capture_good and capture_bad commands used in the tool indicate the start of a build process and the tool will monitor compilations using proxies. The proxies are programs that are used in place of the compiler and their purpose is to examine the compiler invocation, save information needed for the search process, and ultimately invoke the compiler. They are created by the tool and are used in place of the compiler while building (by setting CC/CXX when using a makefile, for example).

The proxies will record compilation steps until a good or bad command is issued. This means that building and testing can be arbitrarily complex, providing that they are not distributed across multiple systems and all object files are distinct. Files can be generated and then compiled, linked into libraries and archived as part of multiple builds; as long as the proxies are used in the compilation steps for the object files, they will be considered in the search.

In its search, the Bad File Finder tool considers only the parts directly shadowed with proxies. For example, consider a build where several libraries are built and then linked together as part of a single build. The tool will work with all of the source files, combining them into different versions of the built libraries as it searches for the problematic file. However, if the libraries are built by different build processes outside of the tool (that is, not using the proxies), it will not consider the files comprising those libraries in the search. The boundaries of what constitutes a build is controlled by the initial capture_good or capture_bad command and the corresponding good or bad command issued by the user.

Testing can also be arbitrarily complex, although any use of the compiler in the testing process should not use the proxies.

How to use Bad File Finder

The Bad File Finder tool is located in /opt/IBM/openxlC/17.1.2/tools and is named bad_file_finder.sh.

Important: Python 3 is required to run the Bad File Finder tool.

Detailed information about the Bad File Finder tool's process, as well as detailed instruction on using all commands and options can be found in the tool's help, which is shown with the -h or --help option.

/opt/IBM/openxlC/17.1.2/tools/bad_file_finder.sh [-h]

Examples on using the Bad File Finder tool can be found in /opt/IBM/openxlC/17.1.2/tools/bad_file_finder/examples.