My editor mentioned that there might be some interest in writing about the OpenPower program, so people who were idly curious might get curious enough to try it. Being a sucker for new hardware I don't actually have to pay for, I decided to sign up. The sign-up process was painless, although it was a couple of days before someone got back to me. On the other hand, my account was created and configured already.

User access to the OpenPower system is through ssh. You provide the OpenPower project administrators with your public key (use ssh-keygen if you don't have one), and they install it. I set up a new key for this project, because it seemed simpler that way:

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/seebs/.ssh/id_rsa): /home/seebs/.ssh/openpower
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/seebs/.ssh/openpower.
Your public key has been saved in /home/seebs/.ssh/openpower.pub.
The key fingerprint is:
62:53:9b:94:89:1a:af:06:38:18:fb:9a:ec:92:40:dc seebs@vash.cel.plethora.net

The "openpower.pub" key is the file forwarded to the OpenPower project admins. They sent me a notice saying my account was configured on the machine "ibmop720.osuosl.org," and I could log in using "ssh -i openpower seebs@ibmop720.osuosl.org." (There was actually a typo in the process preventing it from working the first time; my follow-up query was answered in just over an hour.)

Looking around

For the first time in my life, I sort of wish top(1) used commas in the numbers it reports, because they're frankly a bit large to parse without help:

top - 15:46:39 up 65 days, 59 min,  4 users,  load average: 0.00, 0.00, 0.00
Tasks: 116 total,   1 running, 115 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0% us,  0.0% sy,  0.0% ni, 100.0% id,  0.0% wa,  0.0% hi, 0.0% si
Mem:  32033412k total, 31313732k used,   719680k free,  4494228k buffers
Swap:  1050616k total,      572k used,  1050044k free, 21668944k cached

Yes, that's 32GB of memory. The 65-day uptime is nothing particularly exceptional, but it does testify that the machine isn't crashing at random or anything. In my limited poking about, I have yet to see the system under a discernible load. Loading up a single build process produced about what you'd expect: 12% load. It's a pretty zippy system, and well suited to an open test situation; unless you're doing huge parallel batch jobs, you don't have to worry about the impact of other users on your own work.

Of course, judging system responsiveness over an Internet connection is hard, due to network latency. Actual benchmark testing of a particular algorithm of interest to you, using a representative data set, would tell you what you need to know.

As to the build environment... It's Linux (SuSE Enterprise Server 9, in particular). It has a fairly broad range of packages installed (although no Web server). The few packages I found missing (such as svgalib) are arguably packages which don't make much sense on a server accessed remotely.

If you've done Linux (or any other UNIX®) development before, the environment feels just like home. You get a non-root account on a shared system, which does impose some requirements if you wish to install your own packages, especially if they have dependencies, but that's no trouble.

Back to top

Compiling applications

One of the first things to try on a new system is building some applications. One of my favorite test cases is NetPBM, which is huge and full of interesting features and makes a good test case. I grabbed NetPBM 10.34 from a download server and tried it out. NetPBM is a wonderful test case: It uses some other libraries and has a broad variety of portable number-crunching code. While you might want to hand-roll your own code for really large amounts of image processing, NetPBM serves as an excellent sample set to test with.

NetPBM's "configure" script isn't autoconf, but it works fine. Well, mostly fine; it guesses a few things wrong. The most immediate problem you notice on a build is a failure to link some shared libraries. The diagnostic message speaks for itself:

/usr/lib/gcc-lib/powerpc-suse-linux/3.3.3/../../../../powerpc-suse-linux/bin/ld:
Error: The symbol `pm_error' has a R_PPC_REL24 relocation, that means 'libpm.o' was
compiled without -fPIC.

This is an excellent error message. The first obvious response is to skim Makefile.config (the file that holds settings for the build) looking for hints of where that flag should be set. Browsing the makefile turns this up:

# We have seen -fPIC required on IA64 and AMD64 machines (GNU
# compiler/linker).  Build-time linking fails without it.  I don't
# know why -- history seems to be repeating itself.  2005.02.23.

	CFLAGS_SHLIB =
	# Solaris or SunOS with gcc, and NetBSD:
	#CFLAGS_SHLIB = -fpic
	#CFLAGS_SHLIB = -fPIC
	# Sun compiler:
	#CFLAGS_SHLIB = -Kpic
	#CFLAGS_SHLIB = -KPIC

Sure enough, adding "-fPIC," as the error message suggested (and as hinted at by the Makefile's comments about two other common 64-bit architectures), fixes the problem. (Well, adding that, and running "make clean" to get rid of the old object files.) The other likely problems are errors with the svgalib support code (avoided if you entered "none" for the location of svgalib.o; the default is to assume that it's available on Linux systems) and X11 quirks, resolved by giving "/usr/X11/lib/libX11.so" as the location for libX11.so.

Installing netpbm into your home directory is easy enough, but after you do this, the netpbm binaries can't necessarily find their shared libraries. The solution is to put your library directory in the environment variable LD_LIBRARY_PATH.

What to do when you aren't root

Users who are in the habit of installing software only with root privileges might find it a bit weird having to install things without them. In general, the best way to handle this is to just treat your home directory as another prefix, like /usr, or /usr/local, with its own bin and lib directories. For applications that use autoconf, use "configure --prefix=$HOME." Set the environment variable LD_LIBRARY_PATH to "$HOME/lib"; if you're using bash, for instance, add "export LD_LIBRARY_PATH=$HOME" to your .profile. This will handle most of the programs out there.

In general, you shouldn't need root access to install and use support libraries. There are specific programs and libraries that genuinely require root access, although they're rare. For instance, you can't build Apache's suexec program without root privileges. However, if all you're doing is building a private Web server to run under your own user ID, you don't need suexec anyway.

Installing software without root privileges is a bit disconcerting for users with a long history of desktop Linux and server admin, but it's no big deal for most software.

Back to top

What's a long long?

The POWER5 architecture is natively 64 bit. Unfortunately, not all software copes with this elegantly, and in fact, the default compiler options generate 32-bit code. The issues of having both 64-bit and 32-bit versions of many libraries on the same system, for linking with both 64-bit and 32-bit binaries, may seem rather elaborate, but with the popularity of AMD's 64-bit extensions to the x86 processor architecture, it's a situation with which developers are now fairly familiar. Being able to support old 32-bit code natively simplifies life for sysadmins. It does make things a little more complicated for developers.

Long ago, Henry Spencer warned C programmers to forswear the heresy that "all the world's a VAX." In recent years, more at issue has been the belief that "all the world's a 386." In particular, C developers had about a 20-year window during which the majority of the target platforms people used to learn C on were 32-bit systems where the largest native type was 32 bits, the pointer type was 32 bits, and the most natural word size for int was, you guessed it, 32 bits.

When 64-bit architectures started showing up, a problem arose: A lot of code depended on long staying 32 bit, and some code depended on long being the largest type. Systems with native 64-bit types often went to a model called "LP64," where the long and pointer sizes were both 64 bit. Systems with native 32-bit types often kept both long and pointer sizes 32 bit, but introduced a new type, "long long," which was 64 bit. This type, like 32-bit types on early 16-bit hardware, is actually handled by elaborate code generated by the compiler. For more about the history of C types, see this recent C types article on developerWorks.

What this means for you as a developer is that you might have some surprises on 64-bit hardware. Some programs simply don't work in a 64-bit environment yet, because their developers made assumptions about what "all" architectures would be like. (This isn't Power-specific; the same programs generally fail dramatically in x86_64 environments, too.) The default build options, even on 64-bit POWER systems, are for old-style 32-bit applications.

Building for native 64-bit support does not necessarily improve application performance. It's mostly useful because it provides for larger address spaces, for those applications that need over 4GB of data. However, all the 64-bit objects take up more space, more cache space, more memory bandwidth... In short, it takes more work to process them.

This leads to the surprising result that, in some cases, native 64-bit code runs slower than native 32-bit code, on the same processor! Although 64-bit processors typically have wide enough data busses to keep up fairly well, there's still a twofold increase in data to be moved when you switch to 64-bit data. The recent Standards & Specs article on bus architectures discusses some of the issues; raw bandwidth and latency are two of the major influences on how much it matters.

Overall, 64-bit mode tends to hurt performance a tiny bit. On the other hand, if you're working with a huge data set (data mining, or database operations), having access to a much larger data cache may quite easily make up for this. When we talk about the performance "penalty" of switching to 64-bit objects, it's not even within orders of magnitude of the performance you hit the first time you need to swap to disk because a 4GB address space just isn't good enough.

Using 64-bit mode can offer some benefit when doing a lot of integer math on objects too large to be reliably represented in a 32-bit type; the native 64-bit arithmetic instructions are faster than the emulation built around 32-bit operations.

Still, in practice, the majority of developer time on this issue will go into getting the right versions of all the libraries. The OpenPower systems have both 32-bit and 64-bit versions of the installed libraries.

Compiling NetPBM for 64 bit

A bit of a disclaimer is in order: I am not at all sure that my way of doing this was the best way. I just poked around until it worked. The NetPBM build system worked nearly flawlessly in 32-bit mode. In 64-bit mode, it was a fair bit tougher. The first thing to do was just to put the compiler and linker in 64-bit mode; the gcc manual covers this. Adding "-m64" to CFLAGS was trivial. Not so trivial was figuring out how to supply a linker flag to be used in all cases; I finally settled on changing the LD definition to be "$(CC) -m64." That got me as far as the compiler being unable to find the right (64-bit) versions of support libraries. Other packages might make this a bit easier.

Many developer support programs have a preference for picking "default" values (such as /usr/lib), so if they can find a library in /usr/lib, they're not going to pursue the question of whether it might also be found somewhere else. So, for instance, I changed a couple of lines in Makefile.config to prefer lib64 versions of the libraries:

TIFFLIB = /usr/lib64/libtiff.so
JPEGLIB = /usr/lib64/libjpeg.so

To make things a little more complicated, a couple of symlinks were missing from lib64. Normally, on Linux, a shared library will have major and minor version numbers, with symbolic links to more general names. For more information on shared library versioning, see this developerWorks Linux zone article on shared library versions. For instance, here are the files for libpng in /usr/lib on the OpenPower system:

$ ls -l libpng.so*
lrwxrwxrwx  1 root root     11 2006-05-30 16:48 libpng.so -> libpng.so.3
lrwxrwxrwx  1 root root     18 2006-05-25 11:06 libpng.so.2 -> libpng.so.2.1.0.12
-rwxr-xr-x  1 root root 213468 2002-08-05 05:35 libpng.so.2.1.0.12
lrwxrwxrwx  1 root root     17 2006-05-25 11:07 libpng.so.3 -> libpng.so.3.1.2.5
-rwxr-xr-x  1 root root 198626 2004-06-30 18:55 libpng.so.3.1.2.5

By contrast, in the lib64 directory, the "libpng.so" symlink is missing. Merely specifying "-lpng" doesn't work; the correct line in Makefile.config becomes "PNGLIB = /usr/lib64/libpng.so.3."

With a couple more tweaks like this, everything except the Scalable Vector Graphics (SVG) transform compiles; the SVG transform relies on an XML parsing library, and the library file is acquired programmatically, by running "xml2-config --libs;" unfortunately, there's no way to tell it to provide the answer for the 64-bit library. Manually overriding this works, though.

The result is a 64-bit build of NetPBM. The file command, as always, is the first way to check the results:

np64/bin/pamtotiff: ELF 64-bit MSB executable, cisco 7500, version 1 (SYSV),
for GNU/Linux 2.4.21, dynamically linked (uses shared libs), stripped
np/bin/pamtotiff:   ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV),
for GNU/Linux 2.2.5, dynamically linked (uses shared libs), stripped

The new versions are, as it happens, not quite as fast as the 32-bit versions; however, the netpbm library is sometimes used by other applications that wish to export data in a portable format, and a 64-bit version of that library makes this possible for other programs which have a real need to run in 64 bits.

Back to top

What's up next?

This installment introduced the OpenPower project and took an initial look at some of the porting issues you might face migrating to 64-bit POWER systems. The next installment gets into a little more detail on issues to do with porting to a new architecture.

The careful reader may have noticed that, strictly speaking, most of what's been done here could have been done on a PowerPC® system, such as a G5 Power Mac. It could. However, free access to a 64-bit Power Architecture machine isn't something everyone has handy. Also, if you're thinking about a fairly big server, you might be better off testing your code on an eight-way POWER5 rather than on a four-way G5. Last, but certainly not least: You can't beat the price.

Resources

Learn

• Learn more about IBM System p™ (formerly IBM pSeries, and prior to that, IBM RS/6000) servers.
  
  
• The developerWorks series Assembly language for Power Architecture introduces assembly language in general and specifically, assembly language programming for the POWER5.
  
  
• See all three installments in the Taking OpenPower for a spin series.
  
  
• Learn more about shared library versioning.
  
  
• What's a long long? This compendium on C types will tell you more than you ever knew you needed to know.
  
  
• An extremely relevant article on porting Linux apps to a 64-bit environment can be found right here at developerWorks -- as can this intro to 64-bit PowerPC computing.
  
  
• A recent Standards & Specs article on bus architectures discusses bandwidth and latency, among other things.
  
  
• On the subject of "all the world's a vax," commandment #10 in this amusing list was very prophetic.
  
  
• Find some more very interesting (slightly dated) low-level interview detail on POWER5 from Ars Technica.
  
  
• Apple extols 64-bit PowerPC computing on its Developer Connection site.
  
  
• The IBM Semiconductor Solutions Technical Library hosts a wealth of information -- from specifications and user manuals to product briefs and errata and much more.
  
  
• Keep abreast of all the Power Architecture-related news that's fit to print: subscribe to the Power Architecture Community Newsletter.
  
  

Get products and technologies

• See all Power-related downloads on one page.
  
  

Discuss

• Participate in the discussion forum.
  
  

About the author

Peter Seebach first tested his code for 64-bit readiness on an Alpha in 1999. Still, it never hurts to be extra careful. And yes, it all worked, so booyah.

Rate this article

Comments

Back to top

Table of contents

• Looking around
• Compiling applications
• What's a long long?
• What's up next?
• Resources
• About the author
• Comments

IBM SmartCloud trial. No charge.

Unleash the power of hybrid cloud computing today!

---

Public and private solutions in one trial.

Special offers