Although we'd all like it to be otherwise, the process of porting an operating system to new hardware is hardly an instantaneous one, and it is not always easy. You might run into a number of potential difficulties, especially if you are coming to the problem for the first time.

This article, and likely the next couple in the Testing and measuring the TAMS 3011 series, details my experience porting NetBSD to the TAMS 3011. These articles are not about the finished port, but about the process of developing it. I can only hope you find the hilarious errors as funny as I found them frustrating at the time.

Getting it to compile

The first significant step in getting the kernel working is just getting a kernel with code in it that ought to do something. The existing 405GPr systems are all found in the arch/evbppc directory. (Unless otherwise stated, directory names used in this article are relative to /usr/src/sys/.)

This article continues to use the cross-compile toolset described in the previous article, "Porting NetBSD to the TAMS 3011," hosted on a NetBSD system. In principle, you can do this in other hosting environments, such as Cygwin; it was just more convenient for me to use the system I already had.

Configuration files and file specifications are in arch/evbppc/conf/, include files are in arch/evbppc/include/, and actual code for each platform or family is in its own subdirectory, such as arch/evbppc/obs405/ (for the openblocks 200 and 266 systems), or arch/evbppc/walnut/ (for the IBM "walnut" evaluation board).

Of the various systems, the openblocks266 and walnut look to be the most similar to the TAMS3011. I went with the openblocks266. So, the first thing I had to do was to make a new kernel config file in that directory; I named it TAMS3011. The kernel config file includes (by reference, using an "include" directive) several other files; for instance, the OPENBLOCKS266 config file includes the file std.obs405, which in turn includes the file files.obs405. The difference in usage between these files is a little subtle. The std.system file includes options which are expected to be useful and standard on all of the related systems, such as the EXEC_ELF option, while the files.system file includes a list of specific source modules that that particular port uses. Finally, there's a file named Makefile.system.inc, which is automatically included in the Makefile for building the kernel. So, I copied these files along with the main config file, creating the new files, TAMS3011, files.tams3011, std.tams3011, and Makefile.tams3011.inc.

For a first pass, I simply commented out all of the openbios_* calls, which have to do with extracting configuration data passed in by the OpenBios bootstrap environment. I needed a header file to define things such as the UART frequency, so I created a tams3011.h. With these all in place, I was ready to start compiling the kernel and looking for errors.

Obviously, I encountered a number of errors. The first was simply that the default C compiler on my x86 system couldn't compile for PowerPC®. My admittedly ugly solution was to add a number of lines at the top of the TAMS3011 config file (they could go anywhere, but conventionally options like this go near the top):

makeoptions CC="powerpc--netbsd-gcc"
makeoptions AS="powerpc--netbsd-as"
makeoptions LD="powerpc--netbsd-ld"
makeoptions STRIP="powerpc--netbsd-strip"
makeoptions DBSYM="powerpc--netbsd-dbsym"
makeoptions COPTS="-O2"

This overrides the default native tools with tools from the cross-compile toolchain. This isn't the correct way to do it, but it is a first step, and it works well enough to get the compiler started. It would have worked as well to set these as environment variables, and perhaps been a cleaner solution, but I kept forgetting to reset them.

Back to top

A sea of red flags

The NetBSD kernel is built, by default, with a broad array of warnings enabled. More significantly, it is built with the -Werror flag, which promotes warnings to errors; if you have not eliminated all warnings from your code, it won't compile. When you are doing the initial pass of hacking out some code and replacing it, this is a lifesaver; having the build fail with a warning that a variable is possibly being used uninitialized saves a round trip trying to boot the kernel on a remote system. The warning flags are as follows:

	-Wreturn-type
	-Werror
	-Wall
	-Wpointer-arith
	-Wmissing-prototypes
	-Wstrict-prototypes
	-Wreturn-type
	-Wswitch
	-Wshadow
	-Wcast-qual
	-Wwrite-strings

Some options are disabled: -Wno-sign-compare, -Wno-format-zero-length, and -Wno-main. The latter suppresses the warning generated because the kernel's main() routine does not have a signature normally allowed for a main function in a hosted environment.

Back to top

A question of timing

Sorting out the warnings caught a number of things my first haphazard pass had overlooked, for instance, not having specified the UART frequency. A little background is in order. Serial port hardware is driven by a clock at some fixed frequency, say, 25MHz, or 7MHz, or some other likely number. Some systems have a dedicated I/O clock; others use some exotic divisor of the most convenient clock available to reduce the number of parts needed. A serial port runs at a given baud rate by figuring out which divisor of that frequency will most closely approximate that baud rate. So, to use a UART-based serial port, you need to know at what speed the clock is.

My first attempt was to just copy in a value from another port. With this, when I loaded my kernel, I got a string of garbage; it looked like a baud rate mismatch. My second attempt was to copy the value from the Linux® kernel source for this platform. I'll reproduce the code rather than trying to explain it:

	#define BASE_BAUD (((400*1000000)/((0x3C>>1)+1))/16)

I can explain the "400*1000000" part: The 3011's CPU runs at 400MHz. So, I tried this. No luck. In a fit of desperation, I turned to the manual, which said that the UART frequency was, by default, a processor speed over 31. (A footnote to this: If you have to write 31, without actually putting a 3 next to a 1, you could write it as ((0x3C>>1)+1).) So I tried 400 million divided by 31... And it worked. Well, worked is a bit strong. I got a panic within about five lines, but the console was working.

As a side note, when putting magic numbers in embedded systems, be aware of data types. It is not unheard of for a value like this to end up being shoved into a narrow register, and if the value you generate doesn't fit, it will be very surprising indeed. Since you might be starting with a large value (in this case, 400 million), you have a lot of room for typos to create problems. This is easy to forget, especially when you have several levels of nested macros generating such a constant.

If you get unexpected results, you might find it useful to check what the macros are generating. If they're all constants, you can do this with the preprocessor. If there are live calculations, you might put in a small chunk of code that assigns the constant to an object so you can print it from a debugger:

{
uint32_t dummyval;
dummyval = BASE_BAUD;
}

Back to top

The board_info feature, round 1

The board_info feature is a small property database (using the NetBSD kernel's built-in property database support) that is used on a number of 405GPr ports to store data about the system. The first data required are the amount of memory and the processor frequency. I cheated, in the first case, by simply hard-coding the value into the bootstrap functions.

On the OpenBios systems, a hunk of useful bootstrap data is passed into the kernel through a pointer stored in r31. The bootstrap code saves, then copies, this data structure. When the system is far enough up to allow memory allocation, a property database is filled in from this, using the openbios_board_info_set() function. (This is in arch/powerpc/ibm4xx/openbios/openbios.c.) Contrary to my original assumption that this whole chunk of code could be easily omitted, it's used in a few places to obtain configuration information that is needed elsewhere in the kernel. The first two pieces of information used are memory size and processor frequency.

Setting these two items directly is simple enough:

	unsigned int processor_speed = 400 * 1000 * 1000;
	board_info_init();
        board_info_set("processor-frequency",
		&processor_speed, sizeof(processor _speed), PROP_CONST, 0);

This code was essentially copied from the openbios_board_info_set routine, which filled in several other values in roughly the same way. It does, however, have a bug. The bug manifests in the system, after announcing that it's running at 400MHz, by calculating the number of cycles per kernel tick (which are set at 100Hz by default) as zero. Why? Because the clock code thinks the kernel is at 26Hz, and if the kernel is running at 26Hz, and you want 100 interrupts per second, you have less than one cycle per interrupt.

The original OpenBios code used entries in a static kernel structure to store the data that was being stored with board_info_set. The board_info_set macro is a wrapper around the NetBSD propdb interface, and it turns out that PROP_CONST indicates, not that the value can't be changed, but that the propdb code should just store a pointer to the given argument rather than allocating space for a copy. Since the given argument is a local variable, the entry in the database is "whatever that chunk of stack space holds now." Declaring the variable static was the simplest path to fixing this.

With processor-frequency (and later, mem-size) configured, the kernel was surprisingly functional. While a number of devices were working, though, the ethernet controllers were unfortunately not among them.

Back to top

Getting the MAC address (board_info, round 2)

The setting information passed in from OpenBios, on an OpenBios system, includes at least one MAC address, for the emac0 controller found on the 405GPr. This controller does not have a built-in MAC address; the operating system must program it, using information stored in the boot EEPROM. Since RedBoot has already done this, I thought I might be able to skip this step. My first workaround was to replace the error return in the event that no MAC address was available with an attempt to read it off the chip. This came up with 00:00:00:00:00:00, which led me to the discovery that resetting the chip had cleared its stored MAC address. My second workaround was to modify the reset routine to read the MAC address, reset the chip, and restore the address. This was... ugly. But it worked well enough to get the controller to probe, so it was a working step along the road to a correct implementation. (The mere fact that every single line of code touched during this process was to be removed later doesn't mean it wasn't progress.)

The code for writing a MAC address (stored in a six-byte array) to the chip looks like this:

	EMAC_WRITE(sc, EMAC_IAHR, enaddr[0] << 8 | enaddr[1]);
	EMAC_WRITE(sc, EMAC_IALR,
	    enaddr[2] << 24 | enaddr[3] << 16 | enaddr[4] << 8 | enaddr[5]);

The existing code just did this once. My atrocious workaround was to do the analogous read operation to obtain the address at the beginning of a reset, then write the data back; then, back up at the top of the driver, do the read again to see what I wrote. This was not, in developer parlance, a "clean" solution.

The DS83816 ethernet chip on this system also has no local MAC address storage. In fact, most 83815/83816 chips have a local EEPROM (as do most other ethernet MAC chips); this one doesn't. I spent a few minutes trying to debug this, but just then I got e-mail from a TAMS engineer who helpfully mentioned in passing that there's no EEPROM on the 83816 used in this system. What to do? Use the existing configuration. Since my workaround was doing so well on the emac0 controller, I adapted similar code in the sip0 driver. I still couldn't get it to work, and after a bit of fussing, I decided to come back to it later; this turned out to be the absolute best decision I made in the whole networking process.

Back to top

Network root file system and the real-time clock

In fact, with the emac0 interface working, it was quite simple to get network booting running well enough to serve a root file system to the 3011. There were some quirks, most obviously that the real-time clock wasn't working, but I could open a shell and run simple commands. Having learned from my experience with the UART, my first stop was in the documentation. I had, originally, somehow convinced myself that the RTC was part of the CPU; in fact, it isn't, and I had put in a config line for the wrong clock. I put the right clock ("dsrtc0", for the DS1307 real-time clock) in my config file, and that started working immediately. The only configuration needed was the I2C address, which I found in the board documentation for the 3011.

At this point, having conveniently verified that the I2C bus was working, I thought it might be interesting to look at the other device on the I2C bus, the 4KB EEPROM that holds the boot configuration data. This turned out to be surprisingly simple, since there's already a driver for the AT24Cxx chips. The only real problem I ran into was misunderstanding the "addr N" notation used for devices on the I2C bus, which led to an hour or so of trying to debug a chip apparently containing 4k of 0x80 bytes. So now I could read the data. But...

Back to top

Down the rabbit hole

After a bit of poking about, I discovered that the EEPROM was being detected later than the emac0 network controller, so reading the configuration data from it would be a little late to do me any good. The simple solution is to use the config_defer() function in emac0, which registers a callback function to be run later in configuration. I moved the bulk of the emac0 setup into a second function, to be called back later, and had the EEPROM driver grab the configuration information.

This worked, and even eliminated my previous hack of saving and restoring configuration in the reset routine (which was, I think, obviously a bad idea), but it felt pretty ugly. While I was looking into the question of possibly reordering probes, I got the idea to look at how the Linux source does it.

Looking in the Linux source, I found that this information was extracted from a structure, of a type declared in a nearby header file. In fact, oddly enough, the structure was declared in the walnut.h header, which led me to believe that it was the same structure used on the Walnut -- the OpenBios configuration structure that NetBSD already had code for!

I spent a few hours trying to find and extract this data, experimenting with various things. It turns out the whole line of inquiry was mistaken; there are actually multiple data types involved. Most of the IBM4xx boards supported by Linux typedef some structure to type bd_t. However, some of them are using a type defined elsewhere in the kernel, and some define their own, with different layouts. In fact, the tams_moab code was using the RedBoot version of the structure, not the OpenBios structure.

But what really got me confused was the initialization. Here's the initialization for the bd_t object, as found in the Linux source for a few different IBM PowerPC 4xx-series platforms:

	bd_t *bip = (bd_t *) __res;
	bd_t *bd = (bd_t *)&__res;
	bd_t __res;
	unsigned char __res[sizeof(bd_t)];
	bd_t *bip = &__res;
	extern bd_t __res;

Now, these are all from different files, but the basic impression one gets is that there's a reasonably standard type, bd_t, and that there's one of them stored in an external object named __res. What was lacking was a theory of where this __res object was getting initialized, but I eventually found that it was being initialized from a value passed into the kernel in r31. Very, very similar to the OpenBios stuff.

Unfortunately, the whole thing turns out to be a red herring. RedBoot has special code to load information in a particular structure (not the same as the OpenBios one) into memory and pass the address to the kernel when executing a Linux kernel. The idea that I could just grab the OpenBios information and use it was, it turns out, wrong.

Rather than depend on this, I decided to do things the clean way and read the configuration data from the EEPROM early in the boot. This turned out to have its own quirks and excitements. In the next article, I'll get into the details of how it eventually worked, and then move on to interrupt mapping, why bus_space_unmap is your friend, and other things you never needed to know before.

Resources

Learn

• See all of the articles in the Testing and measuring the TAMS 3011 series.
  
  
• Download the datasheet for the DP83816 MAC from National's Web site.
  
  
• Baffled by I2C? The Embedded Systems Academy has a good page describing the protocol and its history.
  
  
• What is OpenBIOS? Is it Open Firmware? Not quite, but almost. The OpenBIOS home page gives you more information.
  
  
• Keep up with the latest news in three ways: find the latest front-page Power Architecture headlines every weekday on the developerWorks Power Architecture home page (in a "standard" size browser window, the news items appear in the lower-right hand corner), subscribe to the RSS feed of the Power Architecture zone editors' blog, or subscribe to the weekly publication of IBM microNews.
  
  

Get products and technologies

• See all Power-related downloads on one page.
  
  

Discuss

• Participate in the discussion forum.
  
  
• Take part in the IBM developerWorks Power Architecture discussion forum.
  
  
• Send a letter to the editor.
  
  

About the author

Peter Seebach has been talking about "porting UNIX" for many years, and was glad to finally get around to it.

Rate this article

Comments

Back to top

Table of contents

• Getting it to compile
• A sea of red flags
• A question of timing
• The board_info feature, round 1
• Getting the MAC address (board_info, round 2)
• Network root file system and the real-time clock
• Down the rabbit hole
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers