What factors are important in real time? What resources do you need for real-time applications? Which are the most demanding applications? Can a general-purpose OS like Linux do real time? developerWorks talked to Paul McKenney of IBM about his personal history in real time, what its real needs are, where it is going, and what IBM technologies are prepared for it.
Paul E. McKenney is a Distinguished Engineer at the IBM Linux Technology Center. He has worked on SMP, NUMA, and RCU algorithms since he came to IBM in the early 1990s. Prior to that, he worked on locking and parallel operating-system algorithms at Sequent Computer Systems. He has also worked on packet-radio and Internet protocols (even before the Internet became popular), system administration, real-time systems, and business applications.
Would you describe some of your early work with real-time applications?
Paul: My first involvement with real time was longer ago than I care to admit, in the early 1980s. My first project was a building control/energy management system, written in Pascal and assembly on a Z80 CPU, no RTOS, just "bare metal." The Z80 was an eight-bit microprocessor manufactured by Zilog that was popular in the late 1970s and early 1980s and is still in use today.
This project required soft real-time response with scheduling deadlines on the order of a second or two. Although that is ridiculously slow by today's standards -- we now have large-scale database benchmarks with two-second response-time requirements -- it was a significant challenge given the blazing 4MHz clock frequency of the Z80, even given the AMD 9511 floating-point processor. The system saved significant money and energy, eventually winning a prize from a utility company. This system could be implemented in Python/Linux on today's PCs and have lots of computing power left over, courtesy of more than two decades of Moore's-Law-induced performance increase.
Later, I worked on a security-access application running on RT-11 on a DEC PDP-11, a dining-hall application on the same system, then finally an acoustic navigation system on BSD 2.8 running on a DEC PDP 11/23, my first in-depth experience with UNIX®. All of these had soft real-time requirements, generally in the one-second range. All could be accommodated easily by a non-real-time Linux running on today's PCs.
In 1986, I took a job as a UNIX systems administrator and said goodbye to real time for a long while.
Did you go back to real-time work?
Paul: In 2001, about a year after I joined IBM, I got involved with the Sony-Toshiba-IBM initiative that's leading up to the Sony PlayStation 3. Of course, game systems have extremely severe real-time requirements. All the gaming systems I have seen recently offer sub-reflex response (and, yes, I do have three teenagers, so I have seen a few video games). Gamers will not put up with jerky response, so one-second response times just will not cut it; milliseconds rather than seconds are required.
So we are in the very interesting state where gaming and entertainment are major factors driving the technology. When was the last time you heard someone say, "Hey, I bought a new PC and Excel really runs a lot faster"? I used to hear that, back in the early '90s.
Where will time slices be in the future? Are the applications driving time slices ever lower?
Paul: For a given application, it's not so much that the requirements are changing; what is really changing is what can be accomplished with a given type of technology. Back in 1982, that building control system was impressive. It had to run on bare metal, as it couldn't spare the cycles for an RTOS. Furthermore, since the Z80 had only a 64KB address space, the application needed every available bit of memory.
Fast-forward to 2005, and the application's requirements haven't changed much. You still need to respond to fans, air conditioners, thermostats, and photocells in a second or two. What has changed is the available technology. You can run Linux, perform 64-bit rather than 32-bit floating point, do disk I/O, use object-based software technology, thanks to gigahertz CPUs and gigabyte memories.
So what are the cutting-edge real-time applications today? Other than gaming, I mean? For instance, what else might Cell be used for?
Paul: Anywhere heavy-duty numerical processing is needed. Gaming, digital media, computer-aided engineering, real-time logistics, computational finance, you name it.
In computational finance, it is not too much of a stretch to imagine Linux clusters running on Cell processor solving Black-Scholes equations in real time, a computational-finance arms race, if you will.
So we're looking at a future where time slices may be measured in nanoseconds?
Paul: There are already applications that require nanosecond response; for example, phased-array radar. Right now such applications use dedicated hardware for that, but maybe a microprocessor fast enough and capable enough could be used for some or maybe all of that application.
Take things like communications. Right now, one could imagine using a high-speed microprocessor to digitally synthesize or decode a US AM broadcast which has frequencies from 550KHz up to about 1.6MHz. However, US FM broadcast operates at frequencies on the order of 100MHz, requiring nanosecond-scale response times.
But how about deterministic response? In your paper, you mention that there are portions of the Linux kernel that are not interruptible and also that there are cases of nondeterministic code. How do you deal with those constraints?
Paul: Let's be very clear on one point before we get too far into that. A lot of times people will tell you it's impossible for a general purpose OS to have deterministic response. In some academic sense, they're right. It's not feasible right now for a general purpose OS to offer all of its services all the time in a deterministic manner.
But that's a narrow viewpoint because a typical real-time application will only require a limited set of services; as long as those services are deterministic or at least meet the time frame and probability distribution required by that application, the response times of the other services are irrelevant. I think that's something the industry as a whole has been very slow to grasp.
So when is Linux or any general-purpose operating system good enough for real-time use? What determines whether it can respond predictably within the time allowed?
Paul: That's an excellent question, but it begs another questions. Namely "which real-time application" and "what services does that application require"?
There are applications that require only user-mode execution. They map a hardware device into the application's user-mode address space and access it directly without any kernel involvement. If the Linux kernel is able to give that application deterministic real-time scheduling latencies, then Linux is good enough for that hard real-time user application. And I believe there are some patches that get the Linux kernel pretty close to the level of real time required by such an application.
Are there any benchmark tools or suites that can be run on a Linux kernel to see what the deterministic results and the "nines" of probability are?
Paul: There are some informal tests that are being used for that, although I don't know whether there are industry benchmarks or certifications for it.
Is Linux ready for real-time applications in which human lives are at risk?
Paul: First off, Linux is already being used for applications where human lives are at risk. One of them is life sciences, where research into genomics and proteomics is expected to lead to novel drugs and medical treatments. If these drugs and treatments do not appear soon enough, some people really will die.
So, in that rather twisted sense, Linux is already being used in applications where human life is at risk.
Yes, the time frames are measured in years rather than microseconds, but human life really is at stake and these people are relying on Linux.
Paul: Yes, I am twisting the question, but I need to make a very important point, namely that the issue is not "hard real time" versus "soft real time," but rather that it is critically important to understand the allowable response times and the required probability of success.
I believe that Linux is ready to handle applications requiring sub-millisecond process-scheduling and interrupt latencies with 99.99+ percent probabilities of success. No, that does not cover every imaginable real-time application, but it does cover a very large and important subset.
So your time slice could fall anywhere along a continuous line and you want to know where it is on that line?
Paul: Yes. Linux is already used in the sense you're thinking; it's also used for a whole bunch of network infrastructure where human lives are at risk because people use networked medical instruments, some of which use TCP/IP to communicate. And a lot of the TCP/IP infrastructure runs Linux.
To go further, if you want to incorporate RTOS capability into the Linux kernel tree, don't you have to make some decisions about what goes into the official kernel? Aren't you at risk of trying to solve everything in one monolithic kernel?
Paul: I think we have to be very careful in deciding what services to offer in a real-time manner. If we were to insist that everything in the Linux kernel must be real time, we would make some really silly choices. Should we refuse to use Linux until it has a deterministic TCP/IP protocol stack? Even for real-time applications that don't use networking? Should we refuse to use Linux until it has deterministic disk I/O? Even for applications that don't use the disk? Of course not.
If the time slice is small enough, is it acceptable in an RTOS to have the kind of data incoherency that might result from not using disks?
Paul: That depends on the application. Let's say we have some real-time application that requires nonvolatile data and 100-microsecond access. What technology do you use to solve that problem? Certainly not today's disk drives.
Perhaps you might use battery-backup CMOS. Perhaps you would instead use a pair of machines with a high-speed low-latency interconnect. It is necessary to look at a specific application and to determine what that particular application requires. Otherwise, you just get lost in meaningless generalities. Some people like working in generalities, but I prefer to solve real problems.
If we put the problem in a Linux context, what can Linux provide as an RTOS?
Paul: The Linux 2.6 kernel, if configured carefully and run on fast hardware, can provide sub-millisecond interrupt and process scheduling latencies with extremely high probabilities of success. There are patches out there that are expected to provide latencies in the tens of microseconds. These patches need some work, but are maturing quickly.
It will likely be quite some time before Linux is completely deterministic; in fact, it is quite possible that Linux never will become completely deterministic. But Linux is already being used for real-time applications and, as it continues to improve, will be put to increasing numbers of real-time uses.
Can you comment on how Linux and real-time applications perform on the PowerPC® architecture? Are there specific applications or classes of applications that stand out?
Paul: One aspect of my reintroduction into real-time was the PowerPC architecture, specifically the Cell processor. Cell is certainly going to be a very high-volume and very exciting member of the PowerPC architecture. Of course, PowerPC has been used for real-time applications for quite some time.
I have read that the Cell processor was designed in part to run an RTOS -- I guess that's obvious, given its gaming focus. What other embedded processors are interesting right now?
Paul: ARM is heavily used for embedded and Linux runs on both 64-bit and 32-bit ARM. There are even SMP ARM parts out there.
It really blew my mind when I first saw that -- a four-core, single-chip ARM, running at 350 and 550MHz, providing 1,440 Dhrystone MIPS, all on 600 milliwatts of power.
Of course, compare that to our PowerPC processor for Xbox which does 700 times as many floating-point operations per second as the four-core ARM does integer operations per second. It has only three cores instead of four, but it does use quite a bit more power. Still, 85 watts is well within range for a consumer device and not that long ago you couldn't buy a supercomputer that could do what PowerPC can now do, regardless of how much power you had available.
But again, the important question is "what does your application need?" If you're running off a battery, the ARM processor we just talked about is high power. You get only a few minutes of that kind of power from a D cell. On the other hand, if you have a wall outlet, 85 watts is trivial -- less than an amp.
Do you see the Cell processor as a breakthrough in real-time processing?
Paul: Potentially. I'm hoping IBM really takes advantage of what it has with this processor because you can get a huge amount of floating point in a small amount of time, doing some really serious real-time control. There are a whole set of applications, currently requiring dedicated hardware, that are likely within reach of the Cell processor.
We do have some ease-of-use issues, and there's a lot of software work that's still needed. We'd like customers be able to write code and have the compiler determine what to give to the SPEs [synergistic processing elements]. Right now, for anything out of the ordinary, the customer has to hand-code SPE assembly language. However, the SPE C compilers are becoming increasingly capable, and many applications will be able to use SPE-optimized library functions.
So we do have a long way to go before our customers can get the full benefit of Cell architecture.
I know you don't have anything to do with how and when we make Cell available -- but if you did, what would you like to see happen?
Paul: Ideally, I'd like to get Cell out there where people can play with it. We have simulators that run on Power and on X86 which we should provide for people to use and experiment with. We need to enable the community.
As for PowerPC, my hope is that we can convince people there should be a development platform based on PS3. Sony did something like that for PS2; you could get a PS2 with a Linux kit some years ago. Hopefully something similar will be available for PS3.
We will still need heavy-duty POWER™-based workstations, since PS3 will not likely meet everyone's needs. Heavy-duty software development would want more disk bandwidth and more memory, for example.
How do you like working with the Linux community? It just keeps growing and growing, doesn't it?
Paul: Hopefully that continues. It's certainly been a very exciting journey for me. It's really been fun for several reasons, not least of which is seeing my work used in so many different places by so many different people. All my previous work has been focused on very small communities.
But when you get to Linux, you have to do things differently. For example, you can't just build a database engine; Linux is more than a database engine. When I'm working on SMP scalability, I must also worry about real-time response, about the desktop, and about single-CPU performance. It's really exciting, because I'm not just repeating the same old database-machine scalability work that I did in the '90s.
|
-
In A comparison of real-time Linux approaches Paul summarizes seven approaches to real-time Linux (originally in an epic 6,000-word post to the Linux Kernel Mailing List).
-
In Linux Kernel Scalability: Using the Right Tool for the Job Paul correlates Moore's Law and SMP software.
-
In Issues with Selected Scalability Features of the 2.6 Kernel Paul calls attention to areas in the 2.6 kernel that could use examining.
-
Many real-time Linux distributions have been ported to PowerPC, including those from IBM Partners like Green Hills, Enea, QNX, LynuxWorks, MontaVista, Wind River, and Denx.
-
Find Application Notes, Tech Notes, User Guides, Product Briefs, and much more, all living in harmony in the informative and well-organized IBM Semiconductor Solutions Technical Library.
- Power resources
- Got news or events you want to share with the Power Architecture Community? Send a note to the editor. See also the article on how to contribute to the Power Architecture Newsletter; and the Newsletter's editorial calendar.
- For more Power content, check out the developerWorks Power Architecture technology zone archives and the IBM Semiconductor Solutions Technical Library.
- Keep up with of all the latest Power news and documentation! Subscribe to the Power Architecture Community Newsletter today.
-
Return to the front page of the Power Architecture Community Newsletter 15 August 2005, or view the current issue of the Power Architecture Community Newsletter.
McLaren Harris is a long-time writer and public relations/communications specialist in the computer industry. He can be reached at mclh@comcast.net.
Comments (Undergoing maintenance)
Table of contents
Local resources
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
