When "Power" customers are looking to use openmpi with the IBM compilers, there is often concern about how to make the pieces work together.
The good news is the IBM compilers will and should work with openmpi. If you do experience problems, post a question on the Linux for Power DeveloperWorks forum:
- http://www.ibm.com/developerworks/forums/forum.jspa?forumID=375
Topics covered
Easy to use out of the box
First, when using enterprise level distro versions (for example RHEL 5.2 or SLES 10 sp2), openmpi is packaged with the distro and can be used as-is with the IBM compilers. There is generally no need to re-build the openmpi binaries which are provided.
A paper is available which describes how openmpi can be used with the XLC/XLF compilers for a workload like SPECmpi2007.
A new paper is now available which describes how openmpi can be used in a small cluster for a workload like Linpack. In particular, the paper provides some clues on how to leverage 16MB large pages by over-riding the malloc invocation buried in the openmpi libraries.
Do I need to re-build openmpi?
On the Open MPI web site ( http://www.open-mpi.org/ ), there is a link to an FAQ page. A common approach used by customers is they go to the FAQ page at http://www.open-mpi.org/faq and search for "xlc", which nicely brings them to an FAQ which describes how to re-build the openmpi package.
http://www.open-mpi.org/faq/?category=building#build-ibm-compilers
This FAQ describes the problem seen with "libtool" and the combination of libtool, openmpi, and IBM compilers.
In general, there is no need to re-build the openmpi libraries with XLC. XLC / XLF programs can take advantage of the packaged distro openmpi libraries.
I do want to re-build openmpi. What are the tricks?
There are various permutations of openmpi versions, Distro versions, libraries, and IBM compiler versions. Here we will begin to elaborate on the pieces tested:
| openmpi |
Distro |
IBM Compiler version |
Re-building hints |
| openmpi-1.3.1 |
SLES 9 sp3 |
IBM XLC Ver 8.0 and XLF Ver 10.1 |
tbd |
Keep in mind there are limitations on which IBM Compiler versions run on each Distro version.
Is there any advantage to re-building openmpi with the IBM XL compilers?
Work in progress.
The build and test matrix...
| openmpi compiled with |
Application compiled with |
Results |
| gcc with Distro |
gcc with Distro |
|
| gcc with Distro |
gcc with Advance Toolchain |
|
| gcc with Distro |
XLC Ver 10.1 XLF Ver 12.1 |
|
| gcc with Advance Toolchain |
gcc with Distro |
|
| gcc with Advance Toolchain |
gcc with Advance Toolchain |
|
| gcc with Advance Toolchain |
XLC Ver 10.1 XLF Ver 12.1 |
|
| XLC Ver 10.1 XLF Ver 12.1 |
gcc with Distro |
|
| XLC Ver 10.1 XLF Ver 12.1 |
gcc with Advance Toolchain |
|
| XLC Ver 10.1 XLF Ver 12.1 |
XLC Ver 10.1 XLF Ver 12.1 |
|
Is openmpi thread-safe?
Work in progress.
Can I leverage XL's openmp (-qsmp=omp) support with openmpi?
Work in progress.
Can I leverage libhugetlbfs with openmpi?
Work in progress.
There is a work-around available today. See paper.
The latest openmpi 1.3.2 has been updated to handle this smoothly. Details to follow.
