Advantages and challenges of adding scheduling to automated testing
A testing automation system can be used for testing software, system hardware, and firmware across multiple software platform. You can use automation frameworks that are designed to perform regression testing to create tasks, execute them, and monitor the test results against different lab resources. These frameworks work for both command line and graphical user interface execution. An important part of an automation system is the process of scheduling a task.
This article gives you a basic understanding of the architecture of the automation system that is integrated with a scheduling mechanism to encourage you to design and develop an automation framework with a scheduling capability. It also gives you an overview of how to develop the scheduling component by using Java language.
Advantages of scheduling in test automation
Scheduling can be viewed as the subsystem of an automation system. Including the scheduling capability in the automation framework enables you to schedule any task at daily, weekly, or monthly intervals and to schedule it to repeat if necessary. You can also use the scheduler to select any lab resource to perform the test regression. This reduces human interaction and executes automatically at the future time specified. The scheduler also helps you manage time and resources for the regression test and helps in improving test performance, thus success rates.
- Gathering information about the execution environment
There is quite often a difficulty in getting the lab resource that is in a proper state to perform the task and available when you need to use it for testing. It is important to check in advance.
- Geographically distributed of teams
Teams that are using the automation system might be working from different countries. Their time zones might be different than the time zone of the system, where the automation server is running. The time and date of each and every schedule must be in synchronization with the time of the automation system.
- Supplying necessary information on the fly
Sometimes, it is necessary to provide some additional information, such as build and sandbox level, at the time of execution. The scheduler should be able to cope up with the changes made by users at the very last moment of execution.
- Working in conjunction with other components of the automation
Scheduling is the subsystem of the automation system, so it should integrate with the rest of the components of the automation system to produce the desired result. Any kind of miscommunication or inadequate coordination among the components might result in a serious hardware problem in the lab resource under test.
- Satisfying each and every team's scheduling needs
Every team has its own automation needs. Some teams might need to execute tasks daily. The challenge is to understand the requirements from all of the teams, and develop a scheduling mechanism that copes up with all of these diverse requirements.
Architecture of the automation system
The test automation system has two major components:
- The user interface
- The graphical user interface
- The command line interface
- The test engine
The graphical user interface (GUI) provides an easy way of managing the tests. It also helps in creating the test regression that runs against different lab resources to test the images installed on them.
The command line interface is an alternative way of executing tests. In this, the users have to do a little more work to create and manage their tests and test regression, because creation and management of the tests and test regression requires frequent user interactions.
The test engine accepts the tests and test regressions form the GUI or command line interface, or both, and runs them against the lab resources.
The components of an automation system are shown in Figure 1. It includes the user interface, the test engine and the system under test. The subsections that follow describe the functionality of these components
Figure 1. Architecture of an automation system
The user interface
This interface can be a web-based GUI or command line. A web-based interface can be developed using any web technology. A command line interface can be useful for applications that don't use browsers. As a part of test automation system, the user interface plays a very important role. It is the place to start with test automation. Different user interfaces comes with different features. Some of them include maintaining the user work areas in a format for teams. These are the main tasks of a user interface:
- Provide a user-friendly interface
- Provide an easy way of creating tests and test regressions
- Provide test execution environments for tests
- Show the test results
- Assist in management of resources
The test engine is the heart of the test automation system, because it runs the tests against different lab resources. It is a type of software that is used to test software, hardware, or a complete system. Selection of a suitable test engine is very important. It depends on the type of tests that you want to automate. Different test engines provide different functionality. For example, IBM® Rational® Build Forge® and the open source STAX engine provide the features of parallel and sequential execution of tests. Sometimes, the test automation framework contains the test engine, which supports the automation that the framework handles. These are main tasks of a test engine:
- Execute (run) tests or test regressions on the applications or images
- Repeat the test
- Collect the test results
- Analyze the test results
- Monitor the execution
- Send email notifications to the testers about any failures
Microcode development for multiple platforms requires collecting source code from the developers and bundling the code into a unit. After the bundle is installed on the lab resource, it requires multiple tests to validate its correctness. Automation of such tests reduces the time that it takes testers to perform their tests, so they can concentrate more on the microcode development.
Obviously, automation needs to reduce the time and effort required for manual interaction with tests. Testing is done either by running scripts or running commands. The scripts can be written in any language. The test engine should be capable of intercepting these script calls and performing the test.
Selection of the framework, along with the test engine, is an important part of the automation process. There are various frameworks available that can be used in the test automation system, such as the open source Software Testing Automation Framework (STAF), or you can create your own.
Many organizations are working on automation by creating their own automation framework and test engine. After you have these two components of an automation system, you can think about using them in an automation process to perform these tasks:
- Create and manage tests and test regression, using the user interface
- Submit the tests to run on the application or image
- Execute the tests on the application or image (test engine)
- Notify the tester for any attention required
- Generate and analyze the test result (test engine)
- Send email to the tester if the test fails
Automation system with scheduling
The automation system with the scheduling capability is shown in Figure 2.
Figure 2. Automation system with scheduler
It includes one more component called the scheduler. The purpose of the scheduler is to identify the type of the task that you want to execute. If it is of the schedule type, the scheduler will instruct the test engine to schedule it; otherwise, it will give it directly to the test engine for execution. You can schedule the execution through the user interface.
Automation framework builds on the scheduling capability
The automation framework, which includes the scheduling feature, might require running the automation server on a system. Users from different teams might be at different geographic locations, which can result in wrong scheduling times. The scheduler should be able to recognize the time zone difference and schedule the tasks according to user requests.
The automation framework can help you create schedules based on the options given in Figure 3. You can also specify the date and time of the schedule and repeat the schedule if necessary. The schedule described in Figure 3 schedules the task daily and continues it for three days. It also shows the dates that it will run the test. The user interface can also provide options to select the system under test and tasks during the preparation of schedule (not shown in the screen capture).
Figure 3. Automation framework with the scheduler UI
Implementing the scheduling feature using Java technology
Java provides the timer framework, which makes it easy for the programmer to schedule a task. This includes java.util.Timer and java.util.TimerTask classes. Timer and TimerTask work together and provide the scheduling capability. Timer is the class that you use to schedule a task. Task should be an instance of TimerTask. To schedule a task, you must create TimerTask. The TimerTask implements a runnable interface that can be used to create a thread of execution. After the task is created, you can schedule it by using Timer instance. The timer uses the following methods to help with scheduling:
- void schedule(TimerTask TTask, long wait)
- void schedule(TimerTask TTask, long wait, long repeat)
- void schedule(TimerTask TTask, Date targetTime)
- void schedule(TimerTask TTask, Date targetTime,long repeat)
- Test engine generates the test result and analyze it
- void scheduleAtFixedRate(TimerTask TTask, long wait,long repeat)
- void scheduleAtFixedRate(TimerTask TTask, Date targetTime, long repeat)
The method clearly tells how to schedule a task. To provide even more flexibility, you can user the observer pattern to provide a wrapper around the one of the scheduling mechanisms. Using this approach, it also provides inter-thread communication, which informs the scheduler by notifying it whenever any new schedule comes into existence. This approach is based on Java technology. You can come up with your own implementation of scheduling mechanism, whichever is best suited for your automation framework.
Figure 4 describes the interaction among the scheduling components. Whenever someone adds a schedule, the scheduler automatically notifies the executor or test launcher component to execute the test. Once the schedule is given, it has to be removed from the scheduled list and allow the next scheduled test to run at the specified time.
Figure 4. Interaction among scheduling components
Canceling and deleting a task
You can use the Timer class in Java to cancel or even delete the scheduled task with either of these methods:
- void cancel() // cancel the timer thread
- int purge () // delete the cancelled task from timers queue
Framework of an automation system with scheduling capability
The following framework description is one of the solutions that can help you understand the working of automation framework. As previously mentioned, the automation system includes the user interface, the test engine, the and system under test. The automation framework contains the following basic components:
- The user interface, which can be web-based
- The database to store the tasks that you want to execute
- A process that continuously checks the database to get the task
- A scheduler that takes the task from the process mention above
- A test engine that execution the task provided by the scheduler
Figure 5 describes the complete interaction among the components of automation system:
- The user adds the task to be executed, with the scheduled date and time and the system under test.
- This information gets stored in the database.
- The daemon runs the process continuously to gets the job from the database, hand it over to the scheduler, and then add information that includes the test of system health.
- The scheduler then forwards the job to the test execution engine, which interacts with the system under test.
Figure 5. Complete interaction among the components of automation system
The other responsibility of the web framework is to get the result of the test and analyze it. The scheduler helps you manage the following aspects of an automation system:
- Test regression, which contains the tasks that you want to execute against any system under test
- System under test
- Time and date of execution of the test regression and repeats, if required
Performance measurement as an added benefit
The automation helps you not only be more efficient with the time that you spend on repetitive testing, but also helps you analyze the results of tests. It also helps you focus more on the microcode development, instead of spending most of the time in testing. Once the developer done with the microcode development, it is the one-click process to perform the overall testing of the build by managing and scheduling the regression test. Therefore, it reduces the human interaction and human errors during the manual testing.
The scheduling capability, on the other hand, reduces the submission of jobs at repeated intervals. The graph in Figure 6 shows the performance improvement of the overall testing of the newest build on a system under test. The bars represent the testing done on a daily, weekly, or monthly basis, which is possible only with the help of scheduling capability of the test automation framework.
Figure 6. Performance measurement bar graphs
To dig deeper into the possibilities and what Rational software offers for testing and for automation, see the Resources section.
- Dig deeper:
- Scheduling recurring tasks in Java applications: Introducing a simple generalisation of the Java language's Timer class by Tom White (developerWorks, November 2003)
- Choosing a test automation framework by Michael Kelly (developerWorks, Nov 2003)
- Automation for the people: Receive immediate feedback with every source code change by Michael Kelly (developerWorks, Nov 2006)
- Advancing toward test automation through effective manual testing by Bob Levy and Dennis Elenburg (developerWorks, Jun 2005)
- Choosing a test automation framework by Michael Kelly (PDF, undated)
- Seven Steps to Test Automation Success by Bret Pettichord (June 2001)
- Effective Software Test Automation: Developing an Automated Software Testing Tool by Kanglin Li and Mengqi Wu (Sybex, 2004)
- Software Test Automation: Effective Use of Test Execution Tools by Mark Fewster and Dorothy Graham (Addison-Wesley Professional, 1999)
- Check into Rational testing and automation software
- Rational Quality Manager, a web-based centralized test management environment that provides a collaborative and customizable solution for test planning, workflow control, tracking and metrics reporting.
- All Rational quality management solutions, a portal page.
- Rational Build Forge, which provides build and release management capabilities to accelerate and streamline software and systems delivery.
- Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
- Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Improve your skills. Check the Rational training and certification catalog, which includes many types of courses on a wide range of topics. You can take some of them anywhere, any time, and many of the "Getting Started" ones are free.
Get products and technologies
- Check the Trials and Demos page for Rational software.
- Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.
- Participate in the discussion forum.
- Join the Rational software forums to ask questions and participate in discussions.
- Share your knowledge and help others who use Rational software by writing a developerWorks article. You'll get worldwide exposure, RSS syndication, a byline and a bio, and the benefit of professional editing and production on the developerWorks Rational website. Find out what makes a good developerWorks article and how to proceed.
- Follow Rational software on Facebook, Twitter (@ibmrational), and YouTube, and add your comments and requests.
- Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
- Connect with others who share your interests by joining the developerWorks community and responding to the developer-driven blogs.