Cross-browser Web application testing made easy

Tools are available for all needs and all budgets

"Test on multiple browsers" has been a mantra ever since there have been multiple browsers to test on. Testing them all—especially these days—is impossible. But you can come a lot closer than you may think. In this article, learn a variety of techniques for cross-browser testing, from the very thorough to the quick and dirty. The choice you make will depend on your resources, but this is an issue you can't ignore.

Joshua Fruhlinger, Editor and Writer, Freelance

Joshua Fruhlinger is a freelance writer and editor. He lives in Baltimore with his wife Amber and his cat Hoagie.



18 December 2007

Also available in Chinese Russian Japanese

We all know—or we all should know—that we need to test our Web applications on multiple browsers and operating systems. But that's a task that's often easier to agree to in the abstract than to actually undertake. When you start thinking about the many combinations of browsers and operating systems that your customers might be using to access your site, a complete suite of cross-browser tests might start to seem like a drain on your development resources.

What? There's more to life than IE?

Developers in the early '00s might have gotten away with only testing on Internet Explorer, but those days are long gone. IE accounts for about 80% of browser visits today. That's a large majority, but not an overwhelming one—certainly no company can afford write off 20% of its customer base out of hand!

You also shouldn't assume that your particular audience has browser habits that mirror the world of Internet users at large. Browser and OS habits vary by interest and country. Many sites aimed at specialized audiences will be more likely to see non-Windows visitors and non-Internet Explorer browsers. For instance, about 60% of the developerWorks site's visitors use Internet Explorer, with about 35% using Firefox. For the Web development zone in particular, the numbers are 51% using Firefox and 41% using Internet Explorer.

Fortunately, a number of tools and tricks make the process a little easier—and cheaper. In this article, you'll see a variety of strategies you can use for testing; you can pick and choose what will work best for you, based on your needs and resources. I'll begin by explaining some of the more thorough ways you can test, but you'll see that there are alternatives if time and resource constraints prevent you from testing as deeply as you might like.

Browser juggling

Bells and whistles

Don't forget that not everyone will have two of the technologies that many Web applications depend on—Flash and JavaScript—activated on their browsers. No matter which of the techniques I discuss in this article you end up using, you should test with both JavaScript and Flash on and off. (Most of the hosted services offer an easy way to toggle these options.) This goes for cookies, too—turn them off, clear them, do whatever you can to simulate the potential state of a browser visiting your site.

JavaScript is of course the basis of Ajax applications. Cross-platform testing of Ajax code can get kind of gnarly; the bottom line is that the closer you can get to the real user experience, with fewer layers of hardware or browser emulation, the better. The trickiness of debugging Ajax apps across browsers and the difficulty of getting them properly indexed by search engines are two of the major impediments to their adoption. See the links in Resources for more information, including a link to developerWorks' own Ajax resource center.

The most obvious, most thorough—and, generally, the most expensive—method for testing across multiple browsers and OSes is to actually test across multiple browsers and OSes. In many ways, the browser is the easiest part of the equation, as most of the heaviest hitters are available for free. Remember, though, that in all likelihood you're going to want to test not just several different browsers, but several different versions of the same browser. As a developer, you almost certainly have the latest and greatest version of the browsers you use on your computer—this is one of the reasons the "test on multiple browsers" mantra got started in the first place—but a quick look at your site's browser statistics will probably show that at least some of your customers don't.

Running multiple versions of the same browser concurrently can be a bit tricky, as the install process for most browsers overwrites any previous versions. Fortunately, there are workarounds. For Firefox, it's generally as simple as saving the executable of each browser under a different name, and then being sure to launch the profile manager the first time you start each version so that profiles aren't shared. Figure 1 shows two versions of the Firefox browser—the latest 2.0.0.10 release and a creaky old 0.7 version—running side by side.

Figure 1. Multiple versions of Firefox running simultaneously
Multiple versions of Firefox running simultaneously

For Internet Explorer, the process is a little trickier. Windows really doesn't want you to do this; to get around its recalcitrance, you need to tweak some registry settings. TredoSoft, a small software firm, has gained a great deal of visibility by releasing a free, handy tool for installing multiple versions of IE and instructions for mucking about with the correct registry keys and DLLs; see Resources for a link (and for links to other tools and expansions on the tips I'll mention here, though such links don't necessarily constitute endorsements from me or from developerWorks).

Test in real life

Not only should you test various browsers with various configurations; it goes without saying (or should) that test machines (or, in a more modest shop, trusted remote user-testers) should have their own connections to the Internet. In other words, don't forget to test a real-world connection, so that you're not fooling yourself by loading images and scripts locally. Throw in any and all real-life factors you can add in—pop-up and ad blockers, personal firewalls, and other software users are likely to have on their machines, which may affect the way your pages are displayed; this will help you to get a better idea of the real end-user experience. The University of Washington's Web Tripwire Toolkit (see Resources) can help you see when weirdness is caused by interference, rather than browser implementations.

Of course, before you can get all of these browsers up and running, you need to figure out where to get them! Microsoft® doesn't offer Internet Explorer 5 for download from its site, for instance, and it isn't obvious on the Mozilla Foundation's page where you can get Firefox 1.0. Fortunately, enterprising Web citizens have created archives of older executables. The volunteer-driven Web community at evolt.org has an extensive collection of old browsers of every flavor you can imagine, from IE to Opera and more, though there are some gaps in the more recent Mozilla releases; those can be made up at file aggregation sites like filehippo.com and its kin, which you can find without too much trouble. And for Apple's Safari, the Multi-Safari project, a one-developer labor of love, has conveniently packaged up earlier versions of Safari such that they can be run concurrently without fuss. (See Resources for links to all of these.)

Saving screen space

One practical problem you might run into while fiddling with multiple browsers is window clutter. While for the most part that's unavoidable, you can at least consolidate things a little bit by using a couple of handy tools. IE Tab is a Firefox plug-in that allows you to run an Internet Explorer instance from within a Firefox tab. It's not emulation—it really uses the IE rendering engine, which means that what you see is what you really get in Explorer, though it also means that it only works on Windows. Figure 2 shows it in action.

Figure 2. Using IE Tab
Multiple versions of Firefox running simultaneously

RightLynx is a plug-in for Internet Explorer and Firefox that pops up a window when you right-click that shows you what the current page looks like in Lynx so that you don't have to keep an active terminal window open; a Web service loads the page in Lynx and then sends the resulting text back to your computer. Both these tools are free of charge and simple to install, and you can find links in the Resources section.

Lynx: So old it's new

You may have done a double-take when you saw Lynx mentioned in the previous section. Lynx (for those of you too young to remember) is a text-based browser first developed in the early 1990s for use on the UNIX® command-line (though there are versions for DOS, and thus for Windows, as well). Figure 3 shows the Web development zone as seen in Lynx.

Figure 3. Lynx in action
Lynx in action

Testing your site in Lynx might at first seem like a pointless exercise—after all, just about any person visiting your site is going to have access to a modern graphical browser. But the stripped-down, bare-bones version of your site that you'll see in Lynx is a good approximation of what your site looks like to one of your most important visitors: the Web crawlers and spiders that index your site for the search engines that will provide much of your traffic. Lynx can be a great tool for making sure that important search terms aren't hidden away behind Ajax widgets.

Virtualization and emulation

Of course, browsers are only half the battle in your testing; you'll probably want to test across multiple operating systems as well. This is of course a much more expensive proposition than just downloading some free browsers onto your development machine. The most thorough way to go about it is obviously to actually have a Windows machine, a Mac, a Linux® box, and physical versions of any other systems you want to test. That's liable to break your Web development's budget for a year, though—but fortunately, there a number of ways to turn one computer into multiple test platforms.

One of the most straightforward ways is to create dual-boot setups, where multiple OSes are installed on a single machine, with a choice being made at startup as to which one to load. Generally speaking, the OSes are all saved on separate hard drive partitions, but so-called LiveDistros are stored on CDs or thumb drives to avoid the hassle of hard drive formats. Dual booting has historically been associated with Linux and other open source distributions, but the advent of x86-based Macs mean that many machines can now dual boot Windows and OS X.

The downside of dual booting is that switching between OSes is a time-consuming process. This is not the case in virtualization, which involves multiple OSes running simultaneously. Once the province of powerful servers, this tool is now well within the reach of anyone with a standard-issue desktop PC. If you have any intention of serving users with more than one OS, you'll want a test machine with some form of virtualization on board. In Figure 4, I'm using Virtual PC on a Power Mac to look at the developerWorks Web architecture zone in four different browsers on two OSes simultaneously.

Figure 4. Four browsers at once: Safari on OS X, Firefox on OS X, Firefox on Windows XP, and Internet Explorer
Four browsers at once: Safari on OS X, Firefox on OS X, Firefox on Windows XP, and Internet Explorer

A number of tools are available, including VMWare and Virtual PC, that make it easy to run Windows and most other x86-compatible OSes on any generic PC. One OS acts as a host, and the others run within the virtualization application. This makes it possible to test your applications on, say, Windows Vista, Windows XP, and the latest Ubuntu distro in adjacent windows on your screen, without rebooting. You'll still have to buy all the different OSes that aren't free, but at least you only need one piece of hardware.

You'll note that Mac OS X wasn't among the sample OSes I listed in the previous paragraph. Unfortunately, OS X won't run on anything but Mac hardware, but there are virtualization options available for the Mac. Thus, if you're looking for the broadest coverage available on a single machine, you may want to consider using an Intel® Mac and one of the two major virtualiztaion products available for it, VMWare Fusion or Parallels Desktop (see Resources for links). With such a setup, you can test OS X, Windows, Linux, and any other operating systems that run on x86 hardware. The downside, of course, is that neither the software nor the hardware is free, or even particularly cheap.

Going mobile

If your Web site or application will be targeted at mobile users—or even if you expect significant numbers of users from mobile devices—obviously your ideal is to test on actual mobile devices. Unfortunately, the mobile device market is much more fragmented than the PC market, making it difficult and cost prohibitive to buy and maintain a huge stable of physical test devices. Fortunately, you can also use a desktop computer to emulate mobile platforms.

Many specific mobile platforms have free SDKs available that include device emulators that run on a PC desktop. For instance, you can download the SDK for Google's Android platform (see Resources for a link). Getting access to the browser is a little clunky—you need to install the Android plug-ins into the Eclipse development environment and then launch one of the sample applications—but once you do, you'll have access to a browser based on WebKit, the open source framework that underlies both Safari and the browser on Nokia's S60 smartphones. Figure 5 shows the browser emulator in action.

Figure 5. The browser in the Android SDK
The browser in the Android SDK

The market for mobile browsers is again rather fragmented, but many of the higher-end smartphones are now using modified versions of their desktop cousins—the iPhone uses Safari, and Opera Mobile is used on many smartphones, and you can do much of your testing for the phone browsers directly on the desktop, though the correlation isn't perfect. (iPhoney is an OS X app that uses the standard desktop version of Safari, but at least gives you a sense of how your site might look in the iPhone's restricted screen space; see Resources for a link to the free download.)

Of course, many phones are still out there that access the Web through the "walled garden" of WAP. Innumerable free WAP emulators are available online that simply run in a browser window if you need to test a WAP site.

In general, the mobile space is still the trickiest to get a really good, broad coverage through in-house testing. But you do have other options, as I'll show you in the next section.

Outsourcing the testing

You may have reached this point in this article and be feeling a certain amount of despair. Perhaps your deadline is tomorrow. Perhaps you are not convinced. Perhaps you are on a strict budget, either time-wise or dollar-wise. Perhaps your IT department frowns upon you installing old browser versions on your computer or tweaking the registry keys. What are you to do? Will you be unable to test on anything other than the latest versions of Firefox and IE?

You do still have one other option: You can use a hosted service to do the testing for you. You supply the URL, and they show what that URL looks like on a variety of platforms. You lose the ability to really whale away on a site from your own test machine, but you get access to quite a broad array of platforms for very little cost—and sometimes for no cost at all.

Browsershots.org exemplifies what you can get from the free version of this sort of service. Figure 6 shows the different options you have available to you (which don't include mobile platforms or Lynx). For each box you check, you get a PNG file with a screenshot of the URL you specify for that browser/OS combination. The process can take a half hour or more, so you'll want to keep it running in the background while you do something else, but you can cover the important bases for free.

Figure 6. Customizing results from Browsershots.org
Customizing results from Browsershots.org

When it absolutely, positively has to be tested overnight—and tested everywhere

If you have the budget to pay, you can get a bit more advanced. For instance, for a monthly fee, Browsercam.com gives you access to a broader range of platforms and browsers—including some mobile platforms—and these pages are dynamic, allowing you to test Ajax apps and other JavaScript-based pages. Some services, like Litmusapp, can integrate with your testing and development workflows, offering versioning support and ways to publish test results to external clients.

These sorts of services are also well suited to the mobile world, where new devices are always popping up faster than most people can keep track. The DotMobi Virtual Developer Lab is a good example of a hosted service that focuses on the mobile space; it offers access to hundreds of different mobile devices and lets you manipulate them in a number of ways. The interface for both is still not the same as dealing with a real live device, but in many cases it's as close as you can expect to get; and as it focuses exclusively on mobile devices, it would serve as a complement to either your own internal tests or another service for desktops.

Conclusions

Ideally, every Web application should be tested to ensure that it will work perfectly on every browser that might access it. But with the fragmentation of the browser market and the increasing importance of the very fluid world of mobile platforms, that's a practical impossibility. Still, you can come closer than you might think. You can use a wide array of tools for cross-platform Web testing. Whether you have the resources for a workstation with several virtualized OSes, or can only run your application through a hosted service, you have the capacity to see how your application will work in different user environments—and you owe it to your users to make it work as well in as many places as you can.

Resources

Learn

Get products and technologies

  • Android SDK: Google's mobile development toolkit is free to download and includes an Eclipse plug-in that provides a mobile browser emulator.
  • evolt browser archive: An incredibly rich archive of older browsers for you to use in downloading and testing. Some gaps from recent years, but includes all versions of Internet Explorer up to 6.
  • Firefox on filehippo.com: Every version of Firefox since 1.0 is available for download here.
  • Multi-Safari: Download multiple versions of Apple's Web browser and run them simultaneously.
  • IE Tab: Run Internet Explorer as a tab in Firefox.
  • RightLynx Lynx Preview Tool: See what your pages look like in Lynx without leaving your graphical browser. Works with Firefox and Internet Explorer.
  • Lynx: Go back to the future with this text-based browser.
  • Virtual PC and VMWare offer generic x86 virutalization options.
  • Parallels Desktop and VMWare Fusion: Popular virutalization products for the Intel Mac.
  • WapTiger: See how WAP pages look right on your PC.
  • iPhoney: See how your sites will look on the coveted iPhone.
  • Browsershots.org: A free hosted cross-platform testing service.
  • Browsercam.com: A paid service that allows you to interact remotely with your Web page on a variety of platforms.
  • Litmusapp: A paid hosted service that can integrate with your development workflow.
  • DotMobi Virtual Developer Lab: This paid service offers access to hundreds of different mobile devices for testing.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Web development on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Web development
ArticleID=278060
ArticleTitle=Cross-browser Web application testing made easy
publish-date=12182007