Manage time and resources better by scheduling automated tests

Scheduling a test in an automation system is an important part of executing the task at specific date and time. It not only helps you be more efficient and use lab resources more effectively, but helps you monitor results. This article describes the considerations, the components of a scheduling subsystem, the functionality that it should include, and how to develop one by using the Java language.

Share:

Vaibhav Rangare (vrangare@in.ibm.com), System Software Engineer, IBM

Author1 photo Vaibhav Rangare is an integration, test, and deployment engineer in the IBM Software Labs, in India. He is currently working on an automation project as part of the Test Tools Development team. He received his MTech in Computer Science and Information Technology from IIT Kharagpur India in 2003.



21 June 2011

Also available in Chinese

Advantages and challenges of adding scheduling to automated testing

A testing automation system can be used for testing software, system hardware, and firmware across multiple software platform. You can use automation frameworks that are designed to perform regression testing to create tasks, execute them, and monitor the test results against different lab resources. These frameworks work for both command line and graphical user interface execution. An important part of an automation system is the process of scheduling a task.

This article gives you a basic understanding of the architecture of the automation system that is integrated with a scheduling mechanism to encourage you to design and develop an automation framework with a scheduling capability. It also gives you an overview of how to develop the scheduling component by using Java language.

Advantages of scheduling in test automation

Scheduling can be viewed as the subsystem of an automation system. Including the scheduling capability in the automation framework enables you to schedule any task at daily, weekly, or monthly intervals and to schedule it to repeat if necessary. You can also use the scheduler to select any lab resource to perform the test regression. This reduces human interaction and executes automatically at the future time specified. The scheduler also helps you manage time and resources for the regression test and helps in improving test performance, thus success rates.

Challenges

  • Gathering information about the execution environment
    There is quite often a difficulty in getting the lab resource that is in a proper state to perform the task and available when you need to use it for testing. It is important to check in advance.
  • Geographically distributed of teams
    Teams that are using the automation system might be working from different countries. Their time zones might be different than the time zone of the system, where the automation server is running. The time and date of each and every schedule must be in synchronization with the time of the automation system.
  • Supplying necessary information on the fly
    Sometimes, it is necessary to provide some additional information, such as build and sandbox level, at the time of execution. The scheduler should be able to cope up with the changes made by users at the very last moment of execution.
  • Working in conjunction with other components of the automation system
    Scheduling is the subsystem of the automation system, so it should integrate with the rest of the components of the automation system to produce the desired result. Any kind of miscommunication or inadequate coordination among the components might result in a serious hardware problem in the lab resource under test.
  • Satisfying each and every team's scheduling needs
    Every team has its own automation needs. Some teams might need to execute tasks daily. The challenge is to understand the requirements from all of the teams, and develop a scheduling mechanism that copes up with all of these diverse requirements.

Architecture of the automation system

The test automation system has two major components:

  • The user interface
    • The graphical user interface
    • The command line interface
  • The test engine

The graphical user interface (GUI) provides an easy way of managing the tests. It also helps in creating the test regression that runs against different lab resources to test the images installed on them.

The command line interface is an alternative way of executing tests. In this, the users have to do a little more work to create and manage their tests and test regression, because creation and management of the tests and test regression requires frequent user interactions.

The test engine accepts the tests and test regressions form the GUI or command line interface, or both, and runs them against the lab resources.

The components of an automation system are shown in Figure 1. It includes the user interface, the test engine and the system under test. The subsections that follow describe the functionality of these components

Figure 1. Architecture of an automation system
Flow diagram

The user interface

This interface can be a web-based GUI or command line. A web-based interface can be developed using any web technology. A command line interface can be useful for applications that don't use browsers. As a part of test automation system, the user interface plays a very important role. It is the place to start with test automation. Different user interfaces comes with different features. Some of them include maintaining the user work areas in a format for teams. These are the main tasks of a user interface:

  1. Provide a user-friendly interface
  2. Provide an easy way of creating tests and test regressions
  3. Provide test execution environments for tests
  4. Show the test results
  5. Assist in management of resources

Test engine

The test engine is the heart of the test automation system, because it runs the tests against different lab resources. It is a type of software that is used to test software, hardware, or a complete system. Selection of a suitable test engine is very important. It depends on the type of tests that you want to automate. Different test engines provide different functionality. For example, IBM® Rational® Build Forge® and the open source STAX engine provide the features of parallel and sequential execution of tests. Sometimes, the test automation framework contains the test engine, which supports the automation that the framework handles. These are main tasks of a test engine:

  1. Execute (run) tests or test regressions on the applications or images
  2. Repeat the test
  3. Collect the test results
  4. Analyze the test results
  5. Monitor the execution
  6. Send email notifications to the testers about any failures

Automation process

Microcode development for multiple platforms requires collecting source code from the developers and bundling the code into a unit. After the bundle is installed on the lab resource, it requires multiple tests to validate its correctness. Automation of such tests reduces the time that it takes testers to perform their tests, so they can concentrate more on the microcode development.

Obviously, automation needs to reduce the time and effort required for manual interaction with tests. Testing is done either by running scripts or running commands. The scripts can be written in any language. The test engine should be capable of intercepting these script calls and performing the test.

Selection of the framework, along with the test engine, is an important part of the automation process. There are various frameworks available that can be used in the test automation system, such as the open source Software Testing Automation Framework (STAF), or you can create your own.

Many organizations are working on automation by creating their own automation framework and test engine. After you have these two components of an automation system, you can think about using them in an automation process to perform these tasks:

  1. Create and manage tests and test regression, using the user interface
  2. Submit the tests to run on the application or image
  3. Execute the tests on the application or image (test engine)
  4. Notify the tester for any attention required
  5. Generate and analyze the test result (test engine)
  6. Send email to the tester if the test fails

Automation system with scheduling

The automation system with the scheduling capability is shown in Figure 2.

Figure 2. Automation system with scheduler
Flow diagram, scheduler between UI and test engine

It includes one more component called the scheduler. The purpose of the scheduler is to identify the type of the task that you want to execute. If it is of the schedule type, the scheduler will instruct the test engine to schedule it; otherwise, it will give it directly to the test engine for execution. You can schedule the execution through the user interface.

Automation framework builds on the scheduling capability

The automation framework, which includes the scheduling feature, might require running the automation server on a system. Users from different teams might be at different geographic locations, which can result in wrong scheduling times. The scheduler should be able to recognize the time zone difference and schedule the tasks according to user requests.

The automation framework can help you create schedules based on the options given in Figure 3. You can also specify the date and time of the schedule and repeat the schedule if necessary. The schedule described in Figure 3 schedules the task daily and continues it for three days. It also shows the dates that it will run the test. The user interface can also provide options to select the system under test and tasks during the preparation of schedule (not shown in the screen capture).

Figure 3. Automation framework with the scheduler UI
Segment of scheduling options selection window

Implementing the scheduling feature using Java technology

Java provides the timer framework, which makes it easy for the programmer to schedule a task. This includes java.util.Timer and java.util.TimerTask classes. Timer and TimerTask work together and provide the scheduling capability. Timer is the class that you use to schedule a task. Task should be an instance of TimerTask. To schedule a task, you must create TimerTask. The TimerTask implements a runnable interface that can be used to create a thread of execution. After the task is created, you can schedule it by using Timer instance. The timer uses the following methods to help with scheduling:

  • void schedule(TimerTask TTask, long wait)
  • void schedule(TimerTask TTask, long wait, long repeat)
  • void schedule(TimerTask TTask, Date targetTime)
  • void schedule(TimerTask TTask, Date targetTime,long repeat)
  • Test engine generates the test result and analyze it
  • void scheduleAtFixedRate(TimerTask TTask, long wait,long repeat)
  • void scheduleAtFixedRate(TimerTask TTask, Date targetTime, long repeat)

The method clearly tells how to schedule a task. To provide even more flexibility, you can user the observer pattern to provide a wrapper around the one of the scheduling mechanisms. Using this approach, it also provides inter-thread communication, which informs the scheduler by notifying it whenever any new schedule comes into existence. This approach is based on Java technology. You can come up with your own implementation of scheduling mechanism, whichever is best suited for your automation framework.

Figure 4 describes the interaction among the scheduling components. Whenever someone adds a schedule, the scheduler automatically notifies the executor or test launcher component to execute the test. Once the schedule is given, it has to be removed from the scheduled list and allow the next scheduled test to run at the specified time.

Figure 4. Interaction among scheduling components
Flow = schedule choices > scheduler > test launch

Canceling and deleting a task

Note:
You can use the Timer class in Java to cancel or even delete the scheduled task with either of these methods:

  • void cancel() // cancel the timer thread
  • int purge () // delete the cancelled task from timers queue

Framework of an automation system with scheduling capability

The following framework description is one of the solutions that can help you understand the working of automation framework. As previously mentioned, the automation system includes the user interface, the test engine, the and system under test. The automation framework contains the following basic components:

  • The user interface, which can be web-based
  • The database to store the tasks that you want to execute
  • A process that continuously checks the database to get the task
  • A scheduler that takes the task from the process mention above
  • A test engine that execution the task provided by the scheduler

Figure 5 describes the complete interaction among the components of automation system:

  1. The user adds the task to be executed, with the scheduled date and time and the system under test.
  2. This information gets stored in the database.
  3. The daemon runs the process continuously to gets the job from the database, hand it over to the scheduler, and then add information that includes the test of system health.
  4. The scheduler then forwards the job to the test execution engine, which interacts with the system under test.
Figure 5. Complete interaction among the components of automation system
Architecture of an automation system

The other responsibility of the web framework is to get the result of the test and analyze it. The scheduler helps you manage the following aspects of an automation system:

  • Test regression, which contains the tasks that you want to execute against any system under test
  • System under test
  • Time and date of execution of the test regression and repeats, if required

Performance measurement as an added benefit

The automation helps you not only be more efficient with the time that you spend on repetitive testing, but also helps you analyze the results of tests. It also helps you focus more on the microcode development, instead of spending most of the time in testing. Once the developer done with the microcode development, it is the one-click process to perform the overall testing of the build by managing and scheduling the regression test. Therefore, it reduces the human interaction and human errors during the manual testing.

The scheduling capability, on the other hand, reduces the submission of jobs at repeated intervals. The graph in Figure 6 shows the performance improvement of the overall testing of the newest build on a system under test. The bars represent the testing done on a daily, weekly, or monthly basis, which is possible only with the help of scheduling capability of the test automation framework.

Figure 6. Performance measurement bar graphs
Human interaction decrease, test success increase

To dig deeper into the possibilities and what Rational software offers for testing and for automation, see the Resources section.

Resources

Learn

Get products and technologies

  • Check the Trials and Demos page for Rational software.
  • Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, DevOps
ArticleID=681626
ArticleTitle=Manage time and resources better by scheduling automated tests
publish-date=06212011