 | Level: Introductory Lewin Edwards (sysadm@zws.com), Design Engineer, Freelance
09 Aug 2005 The Mac mini offers a viable platform for embedded multimedia development. In this article, Lewin Edwards shows how to make efficient, direct use of the framebuffer to display JPEG files and discusses the issues involved in deciding between direct framebuffer access and using the X server as a graphics driver.
In the previous article, you gathered together and installed all the pieces
required to develop your Mac-mini-hosted multimedia application. This
article shows you how to build the first stage of the media player software. It will still handle JPEG images only, but will be easily extensible to handle other media types.
Before I dive into this, however, I'd like to thank everyone who submitted
feedback on the previous article. As a result of your comments, I've
decided to add at least one more article to the end of this series,
dealing with security in embedded Linux™ applications.
Graphical requirements
for an image viewer
We now resume our regular programming. You need to be able to do three things, at minimum, to make a credible still-image viewer appliance:
- Draw bitmapped color graphics to the display.
- Enumerate graphics files available on the hard disk or other
attached storage device (many commercial appliances would use flash media
-- removable or internal -- for storage).
- Decode graphics files.
Of course, other features are also desirable. One almost essential feature
is the ability to scale images to fit in the
screen's physical display area. Additionally, a highly enjoyable cosmetic
bonus is to provide some different transition effects for getting new
images onto the screen.
Hardware graphics support
I'll begin with graphics support. The Mac mini's hardware and the
software bundle you loaded in the last article give you several different
ways of manipulating the screen contents at a pixel level. The canonical
approach would probably be to use X, which is in fact installed on your
hard disk if you followed the instructions in the previous article
exactly. I'll address using X later, but for now just consider that
route as being too complex. Actually, it's not unbearably onerous,
but there are good reasons for describing easier methods first, as you'll see.
A much simpler graphics route than X is to use the kernel framebuffer
driver directly. This conveniently allows you to access video framebuffer
memory directly. The display is pre-initialized into an APA
(all-points-addressable) mode by the kernel; all you need to do is query
the driver to determine the display's characteristics, and map the
framebuffer address into your process' address space.
The oft-quoted disadvantages of using the framebuffer are:
-
Considerable device dependence: Your application needs to be
able to handle all the work of translating from whatever internal color
representation you use into the framebuffer's native memory mapping
format. All the framebuffer driver can really do is tell you what it
expects; it can't handle the manipulation for you.
-
Generally lower performance: Most modern graphics chipsets
have features, such as 3D acceleration, hardware flood-fill, hardware line
draw and so on, that aren't exploited by the simple framebuffer approach.
Native X drivers (where available) are generally quite up-to-date with
support for these acceleration features, so X-based applications will
automatically enjoy any performance benefits. Note, however, that many
embedded platforms don't have hardware acceleration features -- so X is
actually a performance drag on such platforms.
-
Workload: A corollary to the previous point is that the
application developer has the workload of writing or porting any required
graphics primitives -- font rendering, polygon drawing, palette
manipulation, and so on.
When considering the specialized case of embedded applications, the first
bullet point above is often a mere semantic argument. Practically by
definition, in an embedded application, you're choosing (and freezing) the
hardware to be used with your software. The only time you're likely to
worry about device independence is if you're contemplating porting the
code to a new platform with different memory to (sub)pixel mapping.
Likewise, the second point is often inapplicable to embedded development.
In fact, the main reason I want to introduce you to the framebuffer method
before gallivanting off into the world of pre-built GUI code is because
it's the lowest common denominator, and the majority of embedded Linux
devices with a graphical interface are built on top of the framebuffer
device (even if they happen to be running X; hybrid interfaces are quite
common. In fact, this series will show you how to develop a hybrid interface.).
Accessing the framebuffer
The framebuffer device is accessed principally through the device
nodes /dev/fb* (most applications will assume
/dev/fb0). Mode-switching can be accomplished
through ioctls, or from the command line using the fbset utility. This utility, in turn, relies on
the configuration file /etc/fb.modes, which
specifies the timing and default color depth for various video modes.
You can specify a desired boot-time video mode in your PowerPC®-based
system by modifying the yaboot parameter file
to add a video specifier to the kernel options. If you come from an x86
background, you'll be pleased to hear that the yaboot loader is practically identical to LILO in
terms of configuration and usage; the configuration file lives, by
default, at /etc/yaboot.conf, and once you
have edited it, you can transfer the configuration data to the boot
loader itself by running the ybin utility.
To set a specific video mode explicitly at boot time, simply
add append="video=OPTIONS" to the end of the
yaboot stanza that loads your kernel, and
run ybin to commit the change (more on how
to format those OPTIONS in a moment). For example, a yaboot.conf for the Mac mini might look like
this:
Listing 1. A sample yaboot.conf file
boot=/dev/hda6
init-message="Hit TAB for image list, or wait..."
partition=8
timeout=30
install=/usr/lib/yaboot/yaboot
magicboot=/usr/lib/yaboot/ofboot
delay=10
enablecdboot
image=/boot/vmlinux-2.6.10-1.ydl.1
label=linux
read-only
initrd=/boot/initrd-2.6.10-1.ydl.1.img
root=/dev/hda8
append="video=radeonfb:1024x768-16"
|
(For the "shadow" iMac system I set up, the video driver is atyfb, since that machine is based around the
ancient Mach64 video chip. For timing reasons, you should also specify
the video mode as atyfb:vmode:17,cmode:16 on
the iMac -- see below).
You can format the parameters given on the "video=" line using two methods: either video=driver:hresxyres-depth
(where hres is the physical X-resolution, yres is the
physical Y-resolution, and depth is the color depth), or video=driver:vmode:V,cmode:C
(where V is
the video mode number and C is the color depth code). Note that
the vmode codes are arbitrary numbers, originally defined in some
ancient Apple specification for PRAM contents. You'll find them documented
in the Linux kernel at drivers/video/macmodes.h.
Having told you all this, please be warned that you may encounter trouble
if you tinker with the boot-time video parameters, and the reason for this
is pure and simple black magic. Some Mac hardware -- and I'm thinking here
specifically of the iMac backup system I mentioned in the previous article
(see Resources) -- is downright recalcitrant about
its video parameters. This behavior is partly due to various strange
limitations of the integral monitors in Apple's all-in-one systems, and
partly due to unknown issues in the PLL calculation code in the kernel.
By the way, if you happen to be installing on a machine where Linux
can't get a good default video mode, you can use the parameter "video=ofonly" (ofonly means use Open Firmware mode
only). This is conceptually similar to using the VESA BIOS modes on an
Intel®-based PC.
If you want to leave the boot-time video configuration the way the
Linux install set it up, you can simply use the fbset program to change to a more desirable video
mode at your leisure. Begin by adding the following stanza to the end
of your /etc/fb.modes file:
Listing 2. Defining framebuffer modes
mode "1024x768-universal"
geometry 1024 768 1024 768 16
timings 12735 160 32 28 1 96 3
hsync high
vsync high
endmode
|
After you save this file, you can set your system into a 1024x768, 16bpp
video mode using the command fbset 1024x768-universal.
If you're wondering why I developed the above timings, by the way, the
answer is that the integral monitor in the iMac and the LCD monitor I'm
using on the Mac mini both have severe problems locking onto any unusual
syncrates. I couldn't get any of the default (kernel) video modes working
properly on the iMac in particular; the only mode that would work is the
1024x768, 8bpp palettized mode that both machines started in. So I dumped
out that mode's timing values using the fbset
command (with no command-line parameters), changed the bit depth to 16, and
thereby found a happy medium setting that works with both machines.
The code I'll be working with assumes that you have already set the
machine into a compatible (16bpp) video mode, though it doesn't actually
care what the resolution is. In order to ensure that your machine gets
into the right mode automatically on startup, add the following lines
to /etc/rc.d/rc.local, immediately before
the "touch..." line:
fbset 1024x768-universal
setterm -blank 0
The second line, by the way, simply disables automatic blanking of the
screen. If you didn't do this, the display would automatically go blank
after five minutes unless you manipulated the keyboard or mouse.
Scribbling on the screen
Now that you've got the graphics mode set up nicely (well... almost
-- you'll see what I mean at the end of this article), look at
what you need to do in order to write data to the screen. All you really
need is two pieces of information: where the video memory is, and how
it's laid out. You can get this information through a couple of ioctl calls that tell you the visible size, bit
depth, virtual size, and other pertinent information. Refer to graphics.c in the source code archive referenced at
the end of this article. Briefly, the steps are:
- Open the framebuffer device
/dev/fb0.
- Get the "fixed" and "variable" screen information (
fb_fix_screeninfo and fb_var_screeninfo) -- this includes physical
and virtual dimensions, bits per pixel, and physical address of
video memory.
- Use
mmap(2) to map the framebuffer memory
into this process's address space.
- Allocate enough main memory for an offscreen buffer for scratch
purposes.
- Disable the text-mode cursor.
With reference to the last point, if you take a quick peek at the end of
the GR_InitGraphics function, you'll find the
following hack:
Listing 3. Initializing the
framebuffer
// Disable cursor
handle = open(FB_TTY, O_RDWR);
if (0 < handle) {
write(handle, "\033[?1c", 5);
close(handle);
}
|
If you don't have code like this in your system somewhere (note: you could
just as easily use a line like echo -e '\033[?1c'
in your startup files) then you'll be left with a blinking soft-cursor over
the top of your graphics.
The reason the code snippet above is a hack is that it relies on
arcane knowledge that the framebuffer device you're using (fb0) is associated with the console device tty0, which isn't necessarily true. (And, by the
way, nothing prevents this program from being spawned from
a different virtual terminal!) The assumptions made here are, however,
always going to be true, because this is an embedded application
where you can control how the program is loaded and the environment it
will see.
While on the topic of graphics.c and
graphics.h, also observe that I've defined
the unit of measure, as it were, for pixels as a typedef PIXEL, and I've further defined a macro
RGB2PIX that converts an 8:8:8 RGB triad into
the native screen memory format (a 16-bit unsigned
short integer in this case). This provides a rudimentary level
of compile-time device abstraction, which will be useful for you if you
intend to port this code to another platform.
Finding files
Now that you've got the graphics subsystem working, you need to find the
files on your hard disk. This functionality is accomplished by the code
in filescan.c -- principally, FL_Scan(). That function scans for supported files
in the /web directory and populates the
slideshow array. This is a flat array of dynamically-allocated structures.
At the moment, each structure only contains a filename, but this is a
placeholder for later, when you'll add user-defined attributes (slide
duration, and so on) to each image in the slideshow. Note also that
I've implemented a separate function to identify if a given filename is
playable based on a case-insensitive extension match, and to assign an
enumerated type designator. This might seem like an unnecessary frill,
but the feature will be useful later when you support other media
types.
The main loop for this application works as follows:
- Run FL_Scan() to enumerate images. If none found, wait ten seconds
and try again.
- Run through the slideshow displaying each image; pause for ten
seconds between images.
- Repeat indefinitely.
Now to decode some JPEGs! For simplicity's sake, I use the JPEG
library shipped with Yellow Dog. This library is based on the standard IJG
JPEG 6b reference sources and is easy to use. Please note that I wouldn't
normally recommend that you use this library unmodified in a deeply
embedded system, principally because it makes heavy use of dynamic memory
management (malloc/free). However, this application doesn't really qualify
as "deeply embedded," which is why I felt justified in using the standard
library (as well as some of my own dynamic memory allocation).
For a smaller system, you might need to think about the memory issue more
carefully. In a previous life, I worked on several commercially fielded
digital multimedia appliances. I used the exact same Independent JPEG
Group source, but I wrote a small custom memory manager that allocated all
the decompressor's RAM requirements out of a single pool that was reset
every time the codec was invoked. This was all implemented on a rather
unusual system architecture running a proprietary home-rolled operating
system, and based on a PA-RISC microcontroller with 384KB of available RAM
for all requirements (stacks, buffers, global variables, and so on). The Mac
mini system is far less constrained!
Refer to codecs.c for information on how
to invoke the JPEG decompressor. A certain amount of scaffolding is
involved, most of which centers around assembling a setjmp return point
for any errors that occur deep in the bowels of the JPEG library. The
essential steps are documented very well in the example C program linked
in Resources. Note that
reading in a single scanline at a time is officially deprecated for
efficiency reasons. The nature of JPEG encoding is such that the original
file is decoded in eight-scanline chunks (actually, 8x8 blocks), so each
time you call jpeg_read_scanlines, it either
just copies the next scanline out of a pre-decoded buffer, or fetches
the entire next eight scanlines.
In practice, through profiling, I've found that almost all of the decode
and rendering time is spent either in the JPEG math (which you can't
avoid), or in external user-supplied code doing scaling, dithering, and
other functions. That's where all the inner loops are, and fetching one
scanline at a time or eight will not significantly impact overall
rendering speed unless some very unusual circumstances surround your application.
Taking it for a spin
So, now you can drop a couple of JPEG files into the /web directory and run the ibmslides program. On a normal system, you'll see
the images display one after the other (note that they are not scaled
or centered). If you connect through ftp and drop a few more images into
the system, you'll see them appear in the next rotation. Everything is
well and good... or is it? Oh, no, it isn't! On the iMac, the colors
are all wrong!
There's a whole sub-story of frustration and reverse-engineering in there,
by the way, and I'll cut to the explanation without going through all the
experiments I performed. On the old iMac, the framebuffer device reports
that it's running in a standard 5:6:5 16bpp video mode. This is a horrid
lie, however -- it's actually running in a very strange and utterly
nonstandard mode. The most significant bit of each word is used for some
unusual purpose; pixels where that bit is set appear black. The other
color components don't behave as normal either; red, green, and blue cover
the right spread of bits, but the onscreen display intensity isn't
linearly related to what you write into those bits. It almost seems that
the DAC in the Mach64 chip is still in a palettized video mode (using a
CLUT) rather than using direct color mode.
If at first you don't succeed...
How do you fix this? You could either roll up your sleeves and patch the
kernel in a manly fashion, or you could take the weasel's way out and use
something else to initialize the video subsystem. Weasels are
adorable and grossly misrepresented creatures, and here I'm going to
emulate their fine example. I'll use the XFree86 X11 server to
initialize the display!
Actually, in all seriousness, I planned to move this application from
being a vanilla framebuffer program to a hybrid X + framebuffer program
for other reasons anyway -- this annoying phenomenon merely moves up the
schedule a bit.
For the moment, I'm just going to give you a magical solution without
explaining it very much. If you look in the source archive, you'll find a
graphics file called blank.xbm. This is an empty
bitmap file created with the bitmap(1) utility,
and you'll use it to blank out the onscreen cursor.
Before doing anything else, edit /etc/X11/XF86Config, and change the Screen
section to read as follows:
Listing 4. An XF86Config file
Section "Screen"
Identifier "Screen0"
Device "Card0"
Monitor "Monitor0"
DefaultDepth 16
SubSection "Display"
Depth 16
Modes "1024x768"
EndSubSection
EndSection
|
Now copy blank.xbm to /etc (or anywhere else convenient), and then use the
following command line (this assumes that the ibmslides program is in the current directory):
Listing 5. Starting X
X & sleep 10 ; DISPLAY=:0 ; export DISPLAY ; twm & xsetroot -cursor
/etc/blank.xbm /etc/blank.xbm ; xset -dpms s off ; ./ibmslides
|
Presto -- perfect images!
So by this time you have a working PowerPC Linux installation with various
daemons running, and you should now have a basic understanding of how to
use the framebuffer device to implement graphics applications on Linux. In
the next article, I'll explain why this magic works and what's going on.
Then I'll spend more time showing you how to add support for scaling and a
simple scripting language.
Downloads
The downloads for this article are being updated. Please try to download later.
Resources
About the author | | | Lewin A.R.W. Edwards works for a Fortune 50 company as a wireless security/fire safety device design engineer. Prior to that, he spent five years
developing x86, ARM and PA-RISC-based networked multimedia appliances at
Digi-Frame Inc. He has extensive experience in encryption and security
software and is the author of two books on embedded systems development. |
Rate this page
| |
