Many different distributions of Linux are available today. While Red Hat and SUSE LINUX are the two vendors for Linux on POWER solutions supported by IBM, other distributions, such as Gentoo, Debian, and Ubuntu are also becoming popular for Linux on POWER. Application developers are typically interested in ensuring that their application runs on multiple distributions and on different versions of a given distribution. These goals can be achieved by understanding the issues that are involved with binary compatibility. This article defines binary compatibility, discusses considerations for maintaining compatibility, and covers the migration between different releases of Red Hat Enterprise Linux, versions 4 and 5, and SUSE LINUX Enterprise Server, versions 9 and 10. It also includes practices that can ensure compatibility when running an application across multiple distributions.

Table 1 shows the levels of software and supported features available in RHEL4, RHEL5, SLES9, and SLES10 that are covered in detail in this article:

¹You can download the IBM Developer Kit for Linux, Java™ Technology Edition, from IBM. (See Resources for a link.)

Use the flowchart in Figure 1 to determine whether an application is binary compatible on RHEL or SLES.

Back to top

Overview of binary compatibility

Linux on POWER binary compatibility is made possible by conforming to an Application Binary Interface (ABI). An ABI is the interface by which a compiled binary gets access to an operating system and its services. When multiple operating environments support the same ABI, the same binary can run on these multiple environments. More information on the 64-bit supplement to the PowerPC® Executable and Linking Format (ELF) ABI can be found in the "64-bit PowerPC ELF Application Binary Interface Supplement 1.7" (see Resources for a link).

Binary compatibility is the ability to take a binary and run it on multiple environments of a given processor family. These environments could be different versions of the same Linux distribution, or they could be completely different distributions. One example is running a binary that is compiled and run on a POWER5™ processor-based system running SLES10 and executing it on a POWER6™ processor-based system, also running SLES10. Another example is taking a binary that is compiled and run on a POWER5 processor-based system running RHEL4 and executing the same binary on a POWER6 processor-based system running SLES10.

Back to top

Processor compatibility

Processor compatibility is a subject that is closely related to binary compatibility for Linux on POWER. This section covers the compatibility between different 64-bit POWER processors. and the compatibility between the 32-bit PowerPC processor and 64-bit POWER processors.

POWER processor compatibility

The last example under "Overview of binary compatibility" involved running a binary on two different processor types: a POWER5 processor and a POWER6 processor. The POWER6 architecture contains enhancements to the POWER5 architecture while maintaining binary compatibility between the two, which allows you to run the same application on both platforms.

PowerPC and POWER processor compatibility

Binary compatibility between 32-bit applications running in native 32-bit PowerPC environments and 64-bit POWER environments exists. It is also possible to execute 32-bit binaries on native Linux PowerPC environments that were generated on a 64-bit Linux on POWER system. This compatibility is possible because:

• The 64-bit Power Architecture supports the complete 32-bit PowerPC architecture.
• The 64-bit Linux kernel handles 32-bit system calls.
• 32-bit runtime environments contain the necessary 32-bit libraries.
• 64-bit runtime environments contain the necessary 32-bit and 64-bit libraries.

Depending on the development platform, there a few different ways to generate 32-bit and 64-bit Linux binaries:

• The native GNU Compiler Collection (GCC) C compiler on a 32-bit PowerPC platform, such as an Apple Powerbook running Linux, can generate 32-bit binaries that can execute on its native 32-bit platform or on 64-bit Linux on POWER platforms that contain the proper 32-bit user-space libraries.
• The IBM XL C/C++, Version 8.0 and GCC C compilers for 64-bit Linux on POWER can generate 32-bit and 64-bit binary executables, which can execute on either 32-bit or 64-bit runtime environments.
• Cross compilers are available for 32-bit PowerPC Linux systems and 64-bit Linux on POWER systems. These cross-compilers can generate both 32-bit and 64-bit binaries. Regardless of where the binary was built, the 32-bit binary can run on either a 32-bit Linux platform or a 64-bit platform, while a 64-bit generated binary can run only on a 64-bit Linux on POWER system. Crosstool is an example of a cross-compiler (see Resources for a download link).

Table 2 shows how to generate 32-bit and 64-bit Linux binaries for various development platforms:

While the compatibility between 32-bit and 64-bit environments has been demonstrated, this does not mean that distributions officially support this type of compatibility. Red Hat supports 32-bit and 64-bit forward and backward compatibility between RHEL3 and RHEL4, while SLES8 only supports 32-bit forward compatibility when migrating from SLES8 to SLES9.

Table 3 shows the backward and forward compatibility of 32-bit and 64-bit applications on the different releases of RHEL and SLES:

Back to top

Optimizing performance

The 2.6 kernel and the POWER6 architecture contain features that can improve an application's performance. These gains come about because of different libraries, processor features, and updated compilers. Some of the performance gains are immediate without any modification to the application, while other performance gains require a recompile of the source code. Keep in mind that recompiling to gain performance enhancements may compromise binary compatibility on some environments. This section gives some examples of new features of the 2.6 kernel and POWER6 architecture that could improve an application's performance.

NUMA

Non-uniform Memory Access (NUMA) was introduced in the 2.6 kernel for Linux on POWER, and this feature is further optimized in the latest releases of RHEL5 and SLES10. NUMA addresses the problem that arises when certain processors in a system, depending upon where they are on the bus, take longer to reach certain regions of memory than other processors. NUMA reduces the contention for a system's shared memory bus by having more memory buses and fewer processors on each bus. While this is less of an issue on a system with just a few processors, it can provide performance improvements when systems have a large number of processors.

With the Linux 2.6.15 kernel, NUMA optimization improves the performance of workloads that run across large systems (greater than 4-8 processor cores) by specifying that memory may be accessed only by local processors. If a processor is looking for data that is stored in the memory on adjacent cell boards, the Linux 2.6.16 kernel can grab that information and move it over to local memory (which runs a lot faster) and then do the operation required on it without having to restart that operation.

With both the POWER5 and POWER6 architectures being scalable to 64 processors, most applications will benefit from the 2.6 kernel level NUMA support. Applications seeking additional performance gains can use user-land NUMA APIs. RHEL4 ships with user-land NUMA APIs, and more information on how to use these APIs is on the NUMA Group home page (see Resources for a link).

With the Linux 2.6.16 kernel, some changes have been made in the way this shared memory is allocated. If a processor is looking for data that is stored in the memory on adjacent cell boards, the Linux 2.6.16 kernel can grab that information and move it over to local memory (which runs a lot faster) and then do the operation required on it without having to restart that operation.

Compiler enhancements

You might consider recompiling to take advantage of performance gains from the new features in the latest compilers for Linux on POWER. The high-performance compiler IBM XL C/C++ Version 8 is available for base levels RHEL4, SLES9, and SLES10. IBM XL C/C++ Version 9 is available for RHEL5 and subsequent updates as well as SLES10 SP1 and SP2. Version 9 adds performance improvements for POWER6 processor-based systems.

The -qarch and -qtune options are used to optimize performance for their respective architectures. For example, to optimize for POWER6, use -qarch=pwr6 and -qtune=pwr6. Also introduced is the -qtune=balanced option. This option, when used with -qarch=pwr5 (or pwr5x), generates a binary that runs on both POWER5 and POWER6 systems, but with scheduling improvements that should improve POWER6 performance. Version 9 also contains support for AltiVec Vector Multimedia Extensions (VMX), which was initially offered on the IBM PowerPC 970 processor and is now incorporated into the POWER6 processor family using the -qaltivec option.

The GNU Compiler Collection contains compilers for many different languages. Improvements have been made from version 3.3 to 4.1, including POWER6 processor-specific optimizations to its C compiler, gcc. The -mcpu=power6 and -mtune=power6 flags are now supported and result in register usage and instruction scheduling parameters targeted for the POWER6 architecture. There are also VMX vector extensions for the IBM PowerPC 970 and IBM POWER6 processors that can increase the performance of vectorized code. While these optimizations improve performance on their respective architectures, they may compromise binary compatibility for applications running on other platforms.

More information on using the XL C/C++ compiler for Linux on POWER can be found in the developerWorks article "How to use IBM XL C/C++ Advanced Edition V8.0 for Linux on POWER: A guide for GCC users."

SMT

Simultaneous Multi-threading (SMT) provides another possible performance gain when moving to the 2.6 based Linux kernel. SMT is supported on the POWER6 processor and greatly enhances the performance of heavily multi-threaded applications. The POWER6 chip has two hardware instruction threads that are both capable of issuing multiple instructions per cycle, which causes a boost in performance. However, SMT can cause performance degradation in some applications that are not heavily threaded. SMT can be disabled by passing smt-enabled=off to the Linux kernel when booting.

Other 2.6 kernel enhancements

The 2.6 kernel introduced these performance improvements in RHEL4 and SLES9 and continue to improve in the RHEL5 and SLES10 releases:

• Scalability with the capability of 64 SMP POWER6 processors.
• Large page support for memory-intensive applications, allowing 16MB page sizes while still providing the traditional 4KB page sizes.
• Introduction of 64KB page sizes in Red Hat Enterprise Linux 5.
• Virtual memory subsystem enhancements, including a reverse-mapping algorithm that provides improvements in memory-constrained systems.
• Block I/O and asynchronous I/O improvements.

Back to top

Migrating from RHEL4 to RHEL5

Red Hat has maintained a stable ABI from RHEL4 to RHEL5, allowing for a smooth transition of applications between releases. However, to ensure binary compatibility, Red Hat recommends that you link your application interfaces to the core libraries they have defined. This list includes:

• libc, libgcc, libstdc++, libdl, libm, libutil, libcrypt, libz, libpthread, libncurses
• libX11, libXext, libXt, libICE, libSM, libGL
• libgtk, libgdk, libgdk_pixmap, libpango, libatk, libglib, libgmodule, libgthread, libgnomeprint, libgnomeprintui, libgconf, libglade

The core libraries and other compatibility issues are described in detail in the "Red Hat Enterprise Linux 4 Application Compatibility" article (see Resources for a link). Though this document addresses compatibility in RHEL4, the same concepts and ideas can be applied to application compatibility in RHEL5. An application that uses libraries outside of the core set might still maintain compatibility, but further regression testing should be done. Other cases where applications may not retain binary compatible include applications that statically link against any library, depend on kernel level interfaces, or conflict with POSIX standards or the 64-bit POWER ABI definitions.

Not only is the ABI stable between RHEL4 and RHEL5, but many of the 2.6 kernel features found in RHEL5 have also been back-ported to RHEL4. This allows RHEL4 applications already taking advantage of such features as Simultaneous Multi-threading (SMT) and Native Posix Thread Library (NPTL) to gain the performance enhancements included in RHEL5 without having to recompile their source code. These applications will also pick up the performance enhancements that go along with the 2.6 kernel, as described earlier in this article.

There are, however, two areas in which recompiling on RHEL5 could enhance the performance of an application:

• Systems with a large number of processors may find it advantageous to recompile against the NUMA user-land APIs in RHEL5 if this was not already done so on RHEL4. While applications in general will see performance gains from the kernel level NUMA support, additional gains may be achieved by compiling against these user-land APIs. Applications that are processor intensive and access memory frequently will benefit the most since NUMA decreases the time it takes a processor to access a region in memory.
• Other applications might also find performance gains when recompiling with the latest compilers available for RHEL5. These compilers have added flags that take advantage of POWER6 optimizations, mentioned previously in this article.

The risk of compromising binary compatibility must be considered when recompiling your application for performance enhancements. In most cases, though, you should not need to do additional work with your application when migrating from RHEL4 to RHEL5.

Back to top

Migrating from SLES9 to SLES10

The transition from SLES9 to SLES10 should also be relatively smooth. While the NPTL threading model was introduced in SLES 9 instead of using the older LinuxThreads model, NPTL is the only supported thread model in SLES10 due to LinuxThreads support being deprecated in Glibc. The problems that arise due to the changes in the threading model are described under the "Thread model" section, which addresses the concerns when moving from the one thread model to the other. While non-threaded applications that use a smaller subset of common libraries have a better chance of maintaining binary compatibility between SLES releases, do regression testing to ensure quality.

As with RHEL, there are some circumstances when a recompile of the source may benefit an application that is intended to run on SLES10. For example, an application may gain performance boosts by taking advantage of the enhanced compilers available for SLES10. SMT is also a new feature available in SLES10 that will enhance heavily threaded applications.

In general, most nonthreaded applications will be binary compatible between SLES9 and SLES10. However, with the major changes in libraries and kernel versions, a recompile from source is probably the best option for a developer trying to get optimal performance out of their application.

Back to top

Ensuring compatibility across distributions

Writing an application that is portable across multiple Linux distributions can seem like a difficult task, but it can be achieved by following a few simple practices. The latest releases of most distributions typically contain the same level of libraries and kernel versions. Most distributions also use a similar format for configuration and data files.

Both Red Hat and Novell participate in the Linux Standard Base, or LSB, project under the Linux Foundation with the goal of developing and promoting a set of open standards that will increase compatibility among Linux distributions and enable software applications to run on any compliant system even in binary form. In addition, the LSB will help coordinate efforts to recruit software vendors to port and write products for Linux Operating System. The key component of the LSB is the written binary interface specification, which tells developers of Linux applications the standard ways to build and configure their applications. Specifically, the specification lists:

• Common packaging and install guidelines
• Common shared libraries and their selection
• Configuration files
• File placement
• System commands
• Application Binary Interfaces (ABIs) for system interfaces (both application and platform levels)

The following section reiterates some of the LSB specifications and provides some guidelines to follow when writing an application that will be portable across different distributions.

Many Linux distributions include the same ABI and contain a common set of core libraries. However, each distribution's definition of core could vary slightly, which means that regression testing is always necessary when claiming your application is supported on a given distribution.

For example, if you take a closer look at the glibc libraries that are contained in the RHEL and SLES distributions, you see that RHEL4 contains version 2.3.4 and SLES9 is at 2.3.3. The differences in the minor revisions are typically bug fixes with no new added feature. RHEL4, RHEL5, SLES9 and SLES10 also contain similar threading models, so any application that is linked against the common libraries should be able to run on all three of these operating environments. You will also find common libraries in other distributions, such as Gentoo and Debian.

The File Hierarchy Standard (FHS) defines how files and directories should be laid out on common UNIX®-like systems. The FHS should be used if your application relies on other configuration and data files. The main purpose of the FHS is to provide applications a common location to find standard configuration files. More information on the FHS can be found at the Filesystem Hierarchy Standard home page. (See Resources for a link.)

While a statement that guarantees binary compatibility across all Linux distributions typically cannot be made, you can, however, claim support on most current distributions by following these practices.

Back to top

Binary compatibility considerations

While binary compatibility has been defined with regard to an ABI and processor families, there are other compatibility considerations that must also be addressed when determining if an application will run across multiple environments. A few examples include thread models, low-level kernel dependencies, middleware, and core libraries. This section addresses these considerations and describes how Linux on POWER handles them.

Thread model

The threading model in Linux has changed with the release of glibc 2.3 and the transition of the Linux kernel from version 2.4 to version 2.6. The traditional LinuxThreads model used in the glibc 2.2 version and 2.4 kernel has been replaced with the Native Posix Thread Library (NPTL). NPTL was rewritten from scratch and provides significant performance gains for heavily threaded applications. More details on the NPTL specifications can be found in the Red Hat article "The Native POSIX Thread Library for Linux" (see Resources for a link).

SLES8 is based on a 2.4 kernel and contains the LinuxThreads model, which presents a problem when trying to run a threaded application on SLES9 that contains the NPTL threading model. There are two solutions to this problem:

• Make minor modifications to the source code and recompile against the NPTL-based libraries and benefit from the performance enhancements contained in NPTL. More information on migrating from the LinuxThreads model to an NPTL-based model can be found in the LinuxDevices.com article "Migrating applications to the 2.6 kernel and NPTL" (see Resources for a link).
• Set the LD_ASSUME_KERNEL environment variable, which provides backward compatibility to the LinuxThreads model in SLES9. By setting this variable, the linker assumes that a 2.4 kernel is running along with the older LinuxThreads model. Red Hat developer Ulrich Drepper has a detailed explanation of LD_ASSUME_KERNEL (see Resources for a link). However, due to changes in glibc in SLES10, applications that depend on the semantics of the older LinuxThreads model will not work due to the deprecation of the older LinuxThreads model in favor of the newer NPTL in SLES 10. This will also mean that the LD_ASSUME_KERNEL environment variable can only be set to a value greater than "2.6.4." It's best to test to the newer NPTL , the default on SLES 9.

Low-level kernel dependencies

Low-level kernel dependencies are another area of concern when running applications in different operating environments. The movement of information from the /proc file system to the sysfs file system is an example of how an application might not be compatible if it was written against a hard-coded value in one of these file systems. The /proc file system was originally an in-memory file system that allowed user-space access to kernel data structures that contained information, such as system and device status information. This information is being migrated from the /proc file system into the sysfs file system. The /proc file system will still exist, but it will only contain process information.

The storing of the Universal Serial Bus (USB) device list exemplifies the movement of system information from the /proc file system to the sysfs file system. An application that was originally written on a system based on a 2.4 kernel would find this device list in /proc/bus/usb/devices. With the migration to a 2.6 kernel and the sysfs file system, this information has moved to /sys/bus/usb/devices. This movement of information could cause the application to not work properly. While this is not typical, you should be aware of such potential problems at the kernel level.

Middleware and application compatibility

Many applications today are not self-contained, but rather depend on application middleware. Middleware is the piece of software that lies between two applications or between an application and the operating system. Examples of middleware include IBM DB2® Universal Database for Linux, the Java™ Development Kit (JDK), and IBM WebSphere® applications. A developer can determine the version of middleware that their application is supported on, but the middleware provider decides which Linux distributions their product will run on. A list of IBM middleware for Linux is available at the IBM software for Linux Web site (see Resources for a link). Along with middleware, an application may depend on other applications that a distribution ships. Some examples include the Apache web server, database systems, such as mysql and postgresql, and interpreted languages, such as perl and python.

As an example, let's take a closer look at how Java usage could impact an application developer. The Java middleware package receives a lot of focus due to the high number of Java applications that exist because of its platform independence. Not only are there different releases and providers of JDKs, but they also come in 32-bit and 64-bit versions for Linux on POWER. RHEL4 ships with the IBM 32-bit SDK for Linux V1.4.2. RHEL5 ships with both the IBM 32-bit SDK for Linux V1.4.2 and IBM 32-bit SDK for Linux V1.5. SLES9 SP3 ships with the IBM 32-bit SDK for Linux V1.4.2, while SLES10 SP2 ships with the 32-bit version of the IBM SDK V1.4.2 and both the IBM 32-bit and 64-bit SDK for Linux V1.5 . An application developer must be aware of these differences, in case their application depends on features only available in version 5.0.

As another example, the Apache web server is an application that is not considered middleware, but many applications have dependencies on it. Version 2.2 is the latest version of Apache and ships with RHEL5 and SLES10, while RHEL4 and SLES9 ship with version 2.0. If an application contains new features that are only available in Apache 2.2, your application may not be compatible with version 2.0 of Apache that is shipped with RHEL4 and SLES9.

Core libraries

Major changes in core libraries can also present a problem when trying to achieve binary compatibility. Backward compatibility is maintained between major changes in libraries. This allows applications compiled against an older version of a given library to run against a new version of the same library. An application compiled against the latest version of a given library will not run against an older version of the same library if there are major changes between the revisions.

For example, a binary that was compiled on a SLES9 system with a version of glibc at 2.3.3 will also run on a SLES10 system with a glibc version at 2.4 since the 2.4 version is backward compatible. However, the same source file compiled on a SLES10 system will not run on a SLES9 system because of the older level of glibc. This typically presents a problem only when you develop an application on a current distribution and want to support that application on an older version of the distribution that does not supply older compatible libraries.

Back to top

Acknowledgments

I would like to acknowledge Linda Kinnunen for her helpful reviews, and Brent Baude and Matt Davis for their technical assistance and reviews.

Resources

Learn

• Read the "64-bit PowerPC ELF Application Binary Interface Supplement 1.9" (IBM, 2004) for more information on the 64-bit supplement to the PowerPC® Executable and Linking Format (ELF) ABI.
  
  
• More information about Non-uniform Memory Access can be found at the NUMA Group home page.
  
  
• Read "How to use IBM XL C/C++ Advanced Edition V8.0 for Linux on POWER: A guide for GCC users" (developerWorks, December 2006) for more information on using the IBM XL C/C++ compiler for Linux on POWER.
  
  
• "Red Hat Enterprise Linux 4 Application Compatibility" (Red Hat, February 2005) describes the different types of compatibility guarantees that are available to application developers building software for Red Hat Enterprise Linux on one or more hardware architectures.
  
  
• "SUSE Linux Enterprise Application Compatibility" provides information and resources to application developers building software for SUSE Linux Enterprise Server between different release levels and also provides Porting Advisories for software that comes with these releases levels for GCC, Apache, Java(trade/>.
  
  
• The Filesystem Hierarchy Standard has been designed to be used by UNIX® distribution developers, package developers, and system implementers.
  
  
• Read "The Native POSIX Thread Library for Linux" (Red Hat, February 2005) for more details on NPTL specifications.
  
  
• "Migrating applications to the 2.6 kernel and NPTL" (LinuxDevices, March 2004) discusses topics related to migrating existing applications to the 2.6 kernel and the Native POSIX Threading Library (NPTL).
  
  
• Read Ulrich Drepper's detailed explanation of LD_ASSUME_KERNEL.
  
  
• The IBM Middleware on Linux matrix provides information on IBM Middleware availability on Linux.
  
  
• Why Linux on POWER?: The Linux on POWER ISV Resource Center is a comprehensive resource for Independent Software Vendors interested in enabling and promoting their applications for Linux on POWER.
  
  
• The Linux Developer Network (LDN) is the online community for Linux application developers and independent software vendors who want to start or continue their journey developing applications for the Linux platform. The core component of that journey is the tools and information found within the Linux Standard Base (LSB).is a comprehensive resource for Independent Software Vendors interested in enabling and promoting their applications for Linux on POWER.
  
  
• In the developerWorks Linux zone, find more resources for Linux developers, and scan our most popular articles and tutorials.
  
  
• See all Linux tips and Linux tutorials on developerWorks.
  
  
• Stay current with developerWorks technical events and Webcasts.
  
  

Get products and technologies

• Access the IBM Developer Kit for Linux, Java Versions 1.4.2, 5.0 and 6.0.
  
  
• Download the Crosstool cross compiler.
  
  
• With IBM trial software, available for download directly from developerWorks, build your next development project on Linux.
  
  

Discuss

• Participate in the discussion forum.
  
  
• Get involved in the My developerWorks community; with your personal profile and custom home page, you can tailor developerWorks to your interests and interact with other developerWorks users.
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• Overview of binary compatibility
• Processor compatibility
• Optimizing performance
• Migrating from RHEL4 to RHEL5
• Migrating from SLES9 to SLES10
• Ensuring compatibility across distributions
• Binary compatibility considerations
• Acknowledgments
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into Linux on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers