For the UI tests we've implemented them in the unit testing framework that comes with the Dojo toolkit, called DOH (Dojo Objective Harness). This was an easy decision since the application's UI itself is written in Dojo. The DOH tests run within a web browser and can, where required, move the mouse cursor and simulate key presses etc via the use of a Java Applet.
We've demo'd these DOH tests a few times now and we have a pretty compelling story; automated build, install into an app server, run tests, upload test results into Rational Team Concert and ultimately, all viewable in the RTC application's dashboard.
Getting there has not been easy though. Some of the issues we have hit are (and at some point I'll write this up in a bit more detail):
- Running these tests on a real app server instance requires a little Windows level automation (using AutoIt) to do the login before the DOH tests can even be loaded. This is also used to wait for the tests to finish and then save those results to a file.
- The output from DOH is a text based summary of each test which we parse using BeautifulSoup into a XML format that can be uploaded into RTC as if the results had come from JUnit.
- The use of virtual systems to run these tests (as opposed to real physical systems) means that the performance of tests vary significantly. For example clicking a button may take 100ms to update the UI one day, or it may take 10 seconds another. This causes big problems for DOH tests which are usually all written along the lines of do X, wait N seconds then test Y. In particular, although our tests would run perfectly on a real physical system, they would randomly fail on a virtual system due to this performance variability. I ended up changing the design of our tests so that they wait on synchronisation objects (deferreds/promises in the Dojo terminology) instead of using an arbitrary delay of N seconds. With this design the performance of the virtual system did not impact the tests.
- Various bugs in the DOH robot component to do with mouse moves on a virtual system. Namely, the mouse move action is called with a parameter to specify the duration of the action should be, say, 1 second, but on a virtual system it can take anything up to 5 times that before the mouse is in the correct position by which time the next action in the sequence has already run (and failed). For the moment I have coded a workaround, but ideally I would like to push this into the core Dojo source base.
- The DOH tests require a remote desktop session to be permanently open on the virtual system in order to run successfully (specifically, I think it is the DOH robot component has this requirement to handle mouse moves etc). It's not enough to open a connection once and then disconnect it, since that connection seemed to timeout periodically. Instead we used a second virtual system to create a permanent connection to the virtual system running the tests.
- Recent browser updates have caused problems with the DOH robot component, so we have stayed at FF ESR17 instead of upgrading to ESR24. There seem to be fixes for this in progress within the Dojo project.
- Recent JRE updates have prevented the DOH robot component (a Java Applet) from being trusted when run inside of a browser, which means it pops up a confirmation box for every test it runs, breaking a nightly automation run, so we have kept at an older JRE version. As above there seem to be fixes for this in progress within the Dojo project.
- Internet Explorer randomly popups up confirmation dialogs when running DOH robot tests, breaking a nightly automation run, so we have disabled tests on IE for the moment. Hopefully, also covered by the above fixes
At some point I would like to look at The Intern (which I have heard will be the replacement for DOH) to see how much porting effort is required and whether its any better supported than DOH.