Matt Gibble, Marshall Lamb, and Ronnie Sarkar worked on the team that produced Host Publisher, an application that lets customers publish mainframe applications on the Web. The original version of this product ran only on Windows NT and used a proprietary system for recording screen macros. The new version runs on five platforms (with several more in the works) and uses XML for its macro language. We talked with them recently about their experiences.
Given that there wasn't a great deal of Java programming expertise within the development team, why would they choose to create a Java application? "Our goals, from day one, were to make it portable, to make it run within industry standards (like JavaServer Pages and XML technology), and to make it scalable," says Sarkar. The decision to use Java coding was fairly easy; the team wanted the product to run on multiple platforms: Windows NT, Sun Solaris, AIX, AS/400, and OS/390. They had plenty of experience porting applications from one platform to another, but they didn't have the resources or the time to develop separate applications for each platform, or to migrate code that worked on one platform to each of the others. The team also wanted to release the product on all the platforms as close to simultaneously as possible.
Java technology let the team write one set of code that they could maintain and run on all the platforms. Java technology also let the team streamline product testing. Rather than test every panel and every function on every platform, they could pick, as Lamb says, "one platform that we know really well and test the product 100% on that platform. For each of the other platforms, we tested a percentage of the product." This pattern of testing saved time, energy, and money, and gave the team a rigorous test on all the platforms without delaying shipment of the product.
The team not only rewrote the runtime code, they also changed the way Host Publisher records macros for screen definitions and keystroke playback. The first version of the product used a proprietary format. This version uses XML, which provided benefits for Host Publisher's customers, as well as the development team.
Customers can read the macros and edit them. If they want to tweak a macro, they don't have to go all the way back to the Host Publisher Studio; they can just use any text editor.
Also, since Host Publisher's customers are building Web pages, a lot of them are familiar with HTML. Because XML looks very much like HTML, the customers aren't intimidated by it. As Gibble says, whether or not they understand XML, they are more willing to experiment with a file that looks like HTML than a file that looks like Java.
XML is helping the development team do their job faster, too. For one thing, they didn't have to write a parser, and they could still, as Sarkar puts it, "invent a language." He calls XML a perfect fit for this team's needs.
Lamb says that the XML macro code has reduced the time needed to debug problems during development, has been essential for servicing applications the customers create, and has made communicating with customers much easier. Developers can read the application files the customers are working with and, without running the application, can understand what the application is expecting and what it's trying to do. "Along with the error report, I can solve their issues fairly quickly. That's because that format is completely readable, completely editable, and completely portable."
Before the team could make any of these changes, they had to get up to speed on both Java technology and XML.
Whether or not you have Java programming experience, the structures and formats are changing so rapidly that you are likely to have to learn as you go. Gibble says that when he started work on Host Publisher, he had "some basic knowledge of Java itself and the technology as it was a year and a half prior, but by the time I got to this product, a lot of that was either outdated or had changed."
Assuming you have programming knowledge of some kind, Gibble suggests that the best way to get started is just to jump in and start coding. "I had a basic knowledge, and I understood some of the concepts about Java, but when we started development I didn't really have an intimate knowledge of Java and where it stood."
That said, there are some things you can do to speed your transition.
What do you do when you have this somewhat limited knowledge of Java programming language and suddenly you have to develop a Java application that's going to run on five platforms, be translated into 10 languages, and be sold all over the world? You get searching on the Web, and fast.
Gibble says he got most of his knowledge from research. While hardcopy books were often out of date before they hit his desk, the Web provided a real-time reference. As Lamb says, "Java has a really large toolkit base, both within the development environment that we use and ideas and solutions out on the Web. The Java developer public is very vocal and very extensive."
Gibble adds, "It was great to have other developers out on the Internet posting problems that they were having. Being able to search the Internet for things like that made our job a little bit easier." Other resources include newsgroups, such as dejanews (see Resources).
And, as Sarkar says, "Everybody has to learn. If your fundamentals are strong, you can do anything. Be willing to adapt."
Design on paper first. That way, you can clarify what you're looking for while you're searching for what you want. Make the code fit your application, rather than making the application fit the code. "As we were developing and designing the user interface," Lamb says, "we developed on paper first. So we knew what we wanted. Then it was just a matter of matching the technology to it."
Even the best developers don't always know exactly how to implement an idea, whether they are coding in C++ or using the latest Java toolkit. The Host Publisher team didn't waste time reinventing when they could use existing components. For Host Publisher, Lamb says, this was a life saver. "We reused the XML4J Parser from alphaWorks [now part of the Apache XML project], which saved us a lot of time, because we decided early on to use XML as our configuration base. Using XML plus the XML parser from alphaWorks allowed us to save all the time of having to define what all the configuration files looked like, how we were going to parse them, and how we were going to save them."
The developers admit to some stumbles with code they reused. They had to integrate a component product into their code that didn't always provide the functions they needed, so they invented their own solutions. Now that both their product and the component product are working on new releases, they're finding that their solutions are duplicated in the component product's code. As Lamb says, "it was sometimes just a matter of lining up the releases."
Another problem with reusing code comes when your component products are recognizable to your customers. Lamb describes Host Publisher as 'glue' around several component products. Because those products have their own product release cycles, "We have customers who are immediately angry at us for not having released on the newer version of the components. We have a hard time making the public, or the consumers, or even our own marketing folks sometimes, understand that there is a process to putting out a product that demands stability. And stability comes from a large amount of test. You can't just absorb the new stuff that comes out without following that process. So there's always a lag between when the latest and greatest componentry is made available and when we can take advantage of it."
Host Publisher used three levels of technical testing: unit, function, and system. Each developer is responsible for testing the unit he or she develops to make sure that it works the way it is supposed to, and that users can't, for example, enter letters into fields that require numbers.
All the units are then put together, and a separate team tests them to ensure that the product functions the way it is supposed to.
Once the product passes that test, it is passed to a separate organization who tests the product as it will work on the customer's system. The system test team creates many different types of customer scenarios: high-stress environments, multiplatform environments, and environments that mix all kinds of network infrastructure. As Lamb says, "They try to pound the product. They try to break it."
The system test team also tests all the product claims and the bounds of the product (whether it works with browser X and Web server Y, for example). Setting up this kind of environment requires that the testers really know their customers. For new products, that's hard to do. "You rely on prior knowledge, you rely on experience with similar products, you rely on experience with the back-end data sources that they'll be using, you rely on the experience of other products that you're reusing componentry from, and you put together what you think is the best mix of customer environments," Lamb says. "It takes experience with real customers to really understand how our product is being used."
One way to get that real customer information is through beta testing. The team held an open beta, which was advertised on the Internet and allowed anyone to download code and submit feedback. They also held a closed beta, where some key customers were guided through installation and were contacted weekly for feedback. These customers provided a real-life picture of how the product would work once it left the lab. Information from these customers fed back to the system test team, who used it to update their model environments.
The team also ran usability tests, where they invited typical customers into the lab and asked them to use the product and comment on features they'd want to add or would want to see working differently.
Host Publisher was translated into 10 languages. Before the translators ever got the files, the team tested the GUI to ensure that they ironed out potential problems. This testing meant that problems could be corrected once, in the main source files, rather than in each language version. Since potential problems were corrected in advance, the actual translation testing at the end of the development cycle went more smoothly and more efficiently.
Testing the GUI required some creativity on the part of the build team. Because the real estate on a panel is so limited, the longer strings in the translated versions can wreak havoc. Java helps manage this, because a lot of the fields and controls resize automatically, based on the length of the string. Occasionally, though, one control will overlay another.
The team used an automated process called Martian testing to simulate translation. Lamb explains, "You flip a switch during build and it goes through the string tables and adds 40 to 100 percent of additional characters to each string." The Martian testing ended each string with a question mark. "Our build folks actually came up with an algorithm that converted all of the English to pig Latin, so you immediately knew you were looking at a Martian build."
The developers could then go through the Martian panels. If they didn't see the question mark, they probably hadn't allowed enough room for the string.
The time allotted for in-house translation testing comes at the end of the development cycle and only lasts about three weeks. As Gibble says, "Finding out about these problems when the translators actually show up puts a lot of stress on the development team and slows down the process considerably, because a new build is required before any testing can really occur. Flushing them out in advance helped immensely."
Sarkar, working on the server side, says he does a lot of stress testing before the product ever goes to the formal test team. "It's a lot of overtime ... but the only way to deal with the uncertainties of new software is to start early and finish early."
Some surprises from the Host Publisher tests included performance differences on the various platforms. Because Java and the platform-specific JDKs are developed on a few platforms and then ported to many others, those ported platforms always lag slightly behind the latest Java versions from Sun. Lamb says, "That's been our biggest challenge. Not really developing applications for multiple platforms, but getting them to perform similarly ... or at least, how you'd expect them to."
Lamb adds that the Windows NT version of this product written in Java outperforms the original version (which was developed on Windows NT) by a factor of 3. "That's a testament to the power of Java to create high-performing applications. It's just that the discrepancy in performance across the platforms has been difficult to explain, and difficult to overcome."
There were some problems with the leading-edge technology. Sarkar says, "We switched to JDK 1.1.8 very late in the cycle, because we found that under some conditions, there was a memory leak."
XML wasn't all clear sailing, either. Because the product had to be able to handle host data that might be stored in any language, national language support was critical, and wasn't easy to implement. "Some of those national characters get translated into really strange visual characters within an XML file," Lamb says. "And the XML parsers don't always like those special characters. So we had to convert those into escape sequences so that the parser will skip over them."
There were also encoding problems that showed up when the team needed to transfer XML files from platform to platform. Host Publisher relies on FTP to do the file encoding and ensure that the language characters are translated correctly from the Windows code page to the MVS EBCDIC code page. "That doesn't always work predictably and well," Lamb says. "We actually have to rely on a workaround in Java and pre-encode some of these files using the Java character set conversion classes and then transfer the files up."
Overall, he adds, "the benefit of having that open file that's readable, that's understandable, far outweighs any problems we've had to work around."
How do you prepare yourself for change? Design for it.
The Host Publisher team tries to design the product to be as independent of the components and products they use as possible. "We really only rely on their APIs," Lamb says. "As the underlying code changes, those APIs hopefully remain constant, and we have less of a problem integrating the new levels. Inevitably, though, there is a new release of one of those products that completely changes the APIs or completely rewrites the base function that we're dependent on -- those offer the biggest challenge for us. There's not much you can do to prepare yourself for that, except to know that it's coming."
The Java language itself changes frequently. Gibble points out, "You run into problems where method calls have changed, no longer exist -- where classes and class structure change drastically release to release. Especially from where we were, at JDK 1.1.7 to 1.2 and even 1.3." Predict that there will be change between versions and prepare for it. Gibble adds, "There's almost always a workaround -- it just depends on the amount of time and effort you're willing to spend and what the benefit is."
Testing your product thoroughly before release helps you drive out problems created by these changes.
Gibble says figuring out how to use Java to write for multiple platforms is one of the most interesting things about his job. "You can't just write an applet or an application and assume it's going to work on every platform exactly the same. When you think about what Java has done for writing multiplatform applications, it's just amazing. It presents a whole set of new problems and challenges that probably didn't exist before, but even so, the ability to put out a product on four platforms with relative ease has just been incredible."
Lamb says that he actually didn't like writing code in Java as well as he did in C and C++, because he didn't have as much control over the way things worked. "That said, developing and maintaining a product in Java is going to save us a lot of money. Having it work unchanged on multiple platforms is a big boon for us. But also, debugging Java is easier, reusing componentry is easier, supporting the products in the field is easier (because one fix will run on any platform -- we can address, down to the class level, a fix, without having to touch anything else), building the product is faster, and packaging the product for multiple platforms is faster. It's saving this organization a lot of money to have a single-sourced product run on multiple platforms, both from a maintenance perspective and from a development standpoint."
These guys are focused on development; when we asked what the coolest toys they had installed were, they blinked. After a moment, Gibble came up with "the self-voicing kit. Which basically allows a product to have any text that appears on a window spoken." He recently spent a lot of time working to meet some accessibility standards for the product. "It's enlightening to see what it might be like for someone in that situation... just to see how difficult it would be. It gives you a sense of how a product could be designed to meet the needs of customers who take advantage of these requirements."
Since the U.S. government has ruled that all software it purchases after April 2000 must be fully accessible using these types of devices, developers will be seeing more of them in the test environments. Lamb notes that "the technology that has always been there in some of the Win32 APIs, like C and C++, is only just becoming a focus in Java. Java SWING has full accessibility and has accessibility APIs built in. The prior AWT standard does not. It's a testament to the focus on developing fully accessible products, but it's still in its infancy."
Matt Gibble recently left IBM for a job with a startup. Before leaving, he was a senior associate software engineer in the Application and Integration Middleware division. Matt worked for IBM for two years, and holds a B.S. in Computer Science from William and Mary and an M.S. in Computer Science from the University of Virginia. He was the Host Access GUI lead for the 2.1 release of Host Publisher. You can contact him at mgibble@rateintegration.com.
Marshall Lamb is a senior software engineer in the Application and Integration Middleware division at IBM in Research Triangle Park, NC. He's been with IBM for nine years and has a B.S. in Math and Computer Science from Vanderbilt University. He was the chief programmer for the 2.1 release of Host Publisher and is now enjoying his lame duck status. He wrote an article on implementing JavaHelp based on his Host Publisher experience. You can contact Marshall at mlamb@us.ibm.com.
Ronnie Sarkar is a senior software engineer in the Application and Integration Middleware division at IBM in Research Triangle Park, NC. He's been a team lead at IBM for more than five years. He got his B.Tech degree in Electrical Engineering and his M. Tech degree in Computer Science from the Indian Institute of Technology, then earned a Ph.D. in Computer Science from the Ohio State University. He led the server team for the 2.1 release of Host Publisher. You can contact Ronnie at sarkar@us.ibm.com.
- Take a look at the finished product on the Host Publisher site.
- Find out how to do your own Martian testing.
- Get your own pieces of code from alphaWorks.
- Okay, it's shameless self-promotion, but you can get advice, guidance, and code samples on the developerWorks Java and XML zones.
- Check out the XML Parser for Java.
- Read more about coding for accessibility in Building Next-Generation Accessibility into Java.
- Take a tour of The Java Foundation Classes: The New Standard for Java GUI Development (1998).
- Read all about it on dejanews.
- Take a look at the XML project.
- Visit the Sun Java site.
-
The IBM Accessibility Center includes
a detailed Web Accessibility
Checklist for Web developers, plus background
and advice for Java developers and other developers, products
(including Self Voicing
Kit for Java programs, available through alphaWorks), related
links, and more.
Sera Lewis is a developerWorks staff writer and production editor. She believes she is the only IBM employee who will admit to a B.A. in English (from North Carolina State University). Before joining the developerWorks team, she wrote online help and documentation for Host Publisher as an information developer. You can contact her at sera@us.ibm.com.
