Utility programs are compiled or interpreted scripts that support the installation and operations of an application. As important as they are, these programs are often written on-the-fly, as an afterthought to many months of careful development.
Install programs are typically overlooked until the day an application is slated for release, when they're quickly cobbled together to make the production deadline. Operation scripts are frequently ignored until a crisis arises. In both cases, the end result is a scramble to create the needed program -- sometimes causing a delay to the schedule.
This month I introduce five categories of utility program most commonly required for application support. Being aware of these utility programs gives you the opportunity to make them part of your development process and budget now, rather than scurrying to add them at the last minute.
The SHEEP application development team was preparing a new application module for release to production use. The marketing department's new sales-estimating module would enable marketing analysts to compare actual sales with estimated sales. The new module would replace the manually maintained spreadsheet application the marketing analysts were using. Installation was scheduled to happen over the weekend, a mere four days away.
During a final walk-through with the user community, the marketing manager asked, "How will the current sales estimate data be loaded when the application goes live?" The SHEEP architect had an easy answer for that: "We thought you would enter the data, similar to the way you entered the data during user testing." The marketing manager was shocked. "Are you crazy? There are over 4000 rows of data in the master spreadsheet. We only used 40 or 50 rows of data during the tests. We need you to load from the spreadsheet!"
After the meeting, the SHEEP development team agreed that someone would have to write a data-load script to read the spreadsheet and populate the sales-estimate tables. The task was delegated to a lead developer familiar with scripting languages like Perl and VBA (Visual Basic for Applications). The developer worked fast to finish the script and test it out before Friday afternoon. In the end everything worked out: the marketing manager was satisfied with the improved module and the weekend install went off without a hitch.
After the install the SHEEP development team held a meeting to discuss what they had learned from the project. The primary lesson was the importance of considering the application data as well as the programs. The team's mistake was assuming it knew who would load the data and how. As a result the team did not plan for the necessary data load programs, which led to a last-minute scramble to create ad-hoc data load scripts.
The SHEEP application example shows how easy it is to overlook utility programs. The SHEEP development team was concentrating on the primary functionality of the new sales-estimating module. The team's goal was to create a functional module to satisfy the needs of the user community. The team's focus on user satisfaction makes sense -- the user community or application sponsor typically invests in the program and pays for the development. It is appropriate that primary stakeholders receive the most attention in a development process.
Focusing on the user has its downside, though. To be completely usable, an application must incorporate several utility programs, but these programs are typically invisible to the user. Utility programs are usually accessed by support staff, operations personnel, and program installers for tasks like configuration, monitoring, data administration and initialization, and security.
Since utility programs are not used by or even accessible to users, they are likely to overlook them. Not understanding the value of utility programs to the overall application, users and stakeholders often do not allow for them in the budget. Utility programs are considered part of the IT department or support services overhead.
Users aren't the only ones who overlook utility programs, however. Developers do it, too. The basic issue is that utility programs have little to do with the application's core function. Many utility programs are used just once, or very rarely, and most of them are concerned with support rather than primary operations. All of this makes them easy to overlook during planning and development. But doing so is a mistake. For example:
- A program to initialize a database runs just once during program deployment, but your application won't get far without it.
- Utility programs for failure analysis and recovery are rarely used, but you'll know it when you need one. It is challenging to think of failure modes and design appropriate programs in advance.
- Many utility programs exist solely as support operations, which makes them easy to overlook until after an application is deployed. Support programs assist with essential tasks like configuration, monitoring, version management, policy enforcement, and meeting service-level agreements.
One way to avoid overlooking utility programs is to develop a checklist of utility program types, and then use the checklist as a basis for budgeting and planning. In the following sections, I introduce a handful of utility program types. The program types are grouped into five categories:
- Deployment utility programs assist with deployment, installation, and upgrade activities.
- Monitoring utility programs ensure the smooth operations of the application and provide support for diagnosing problems that might occur.
- Management utility programs deal with configuration and the relationship of the application with other applications or system resources.
- Enforcement utility programs ensure the enforcement of security and operational policies.
- Agreement utility programs help ensure compliance with service-level agreements.
This checklist is not exhaustive but it can be a starting point for you to incorporate utility programs into your budgeting and planning activities.
The particular utility programs needed for an application will vary. That is, you do not need to implement every utility program type for every application. Furthermore, how you implement utility programs will vary. You can use standalone scripts or program functionality built into the application. You can use Web-based console or command-line options.
Install programs programmatically deploy the application into the production environment. These programs can take advantage of install program products, such as InstallShield, or install facilities built-in to the operating system, such as the "GO LICPGM" facility on IBM® i5/OS®.
Update programs offer a variation on install programs. The difference is update programs deploy new releases of an already deployed and operational application. As a result, the update program has to work with components and data already in use. In addition, update programs must run quickly to minimize downtime.
Initial data load programs populate an application database with the initial data set. This initial data includes reference data (country codes, currency codes, and other codes), configuration data, customization data, and the starter set of operational data. Often initial data load programs also read data from other sources, such as exports from another application. Many new applications replace manually maintained data, such as within spreadsheets, so the initial data load also frequently retrieves data from these document-based data sources.
Data migration programs are essential when a program update involves data-structural changes. For example, suppose you are releasing version 2.0 of your application and the new version splits a data table into two. In this case you would use a data migration program to retain the data by migrating it from the old single-table design to the new double-table design.
Availability programs monitor the status of the application and ensure it is running. If the application stops, the availability monitor will attempt to restart the application component or notify the operator for manual intervention. Many programs perform overall monitoring of application services, processes, and resources, such as IBM Tivoli® Monitoring. You might need to supplement such products with custom scripts or additional functionality within the application in order to monitor lower level components.
Most programs include some form of logging, which writes messages to a log file. This can be adequate for a small, single-program application. However, when the application grows to encompass many processes and services and many executable files, then it is time to consider an application-wide logging facility.
Left unattended, log files grow large very quickly. Therefore, it is vital to address log file management during the development process. How long is log file data retained? What process removes and discards obsolete log file data? You would implement these management policies using additional utility programs.
Log file analysis programs display log file contents in a more user-friendly manner and enable analysis activities. It is a problem in many log files that the content is compact and often encoded. An analysis program converts log file entries into content that is easier to read. The analysis program might even use a knowledge database that contains a detailed description of the log entry type and history of past solutions.
In most cases, the native operating system or application server facilities for measuring performance are adequate for measuring an application's performance profile. However, if the application has very specific performance requirements, such as ones specified in the service-level agreements, then you will need to build customized performance-measuring facilities.
If your application uses a messaging system, whether using Web services or message queuing, then you know the message path can be a fragile component -- the adapters, channels, services, broker, and communications network are failure points. One way to test the integrity of the message path is to send a synthetic or dummy transaction. One important synthetic transaction is a loopback message; a message forwarded by each component until the message returns to the original sending process. You will need to create utility programs that can introduce these synthetic transactions into the messaging system and capture the reply.
Again, if you use a messaging system, especially one built on a message-queuing product, then you will need to create a program to monitor the dead letter queue. Message-queuing products (such as IBM WebSphere® MQ) reserve one or more queues for recording system errors, undeliverable messages, queue overflow, and other queue-management errors. The dead letter queue program implements the error handling policy; for example, "notify the system operator of any events recorded in these error queues."
Application configuration can be stored in many different forms. Some examples include the system registry, INI text file, Java properties files, and XML files. Since the common form of configuration is a text file, programmers usually use a text editor to update the configuration. While this is adequate in many cases, it can cause problems if there are many configuration elements to maintain. The solution might be to create a customized configuration editor. This approach can also support a distributed environment, where the program propagates changes made to one system's configuration to all peer systems.
Version utility programs display the version of the application and all components that make up the application. These programs help diagnose possible problems caused by an incorrect component version or incompatibility. This approach assumes that all application components are versioned in some consistent manner.
In my experience, every database application, at one time or another, has a problem with corrupted data. The corruption can be a result of program failure, database failure, out-of-sync condition after a database restore from backup, a programmer making direct data changes, and any number of other causes. While data integrity rules implemented using referential integrity constraints, check constraints, and triggers go a long way toward reducing data corruption, they are not perfect solutions. As a result, it is helpful to create data integrity programs that can check the data integrity of the database. These programs run after any system or application failure or programmer intervention.
Related to the data integrity programs are programs to assist with data research. The application's support staff will need to research data anomalies detected by a user. Maybe the task will be to provide the history of certain data so the business user can determine how it got to the current value. Therefore, you might need to create programs or custom queries that look at the data in ways other than the standard reports or views used by the user community.
Even though disk space is relatively cheap and it is easy to create 1000-gigabyte or larger databases, the question is whether this is the best way to handle your application's data. The classic solution, which is still valid today, is to remove from the active database data that is no longer necessary or relevant. This mechanism is data archival. In many cases you cannot just delete the data. It has to be stored for some data retention period -- often set by company policy, industry standards, or government regulations. The corollary is data restore, which loads the archived data back into the application's operational data so you can analyze it.
The data restore program depends on a set of restore-migration programs. What these programs do is update the restored data to the current operational data version. For example, the data structure and content of an archive might correspond to version 2.0 of the application. However, when restored, the current system is running version 5.0 of the application. It is likely that the data structure and semantics have changed between those versions. As a result, you will need to have available programs that can update the archived data from version 2.0 to the latest version 5.0 structure and semantics.
The classic security reset program was a mechanism to reset a user's password. However, powerful security products, such as IBM Tivoli Access Manager, provide a more standardized authentication and authorization.
For modern applications, especially Web applications, often need to create a program to reset session locks and single-session restrictions. My experience with financial intranet applications is the business wants to limit users to a single session -- that is, the user cannot use two browsers at the same time to access the application. User sessions are often limited by a locking mechanism. Now, what happens if a user closes his browser and goes to another system? The server does not detect the closed browser, so the server uses a session timeout to free the lock. Until that session timeout occurs, the user cannot start another session. Therefore, it is helpful to create a session-reset program that the support center can use to release a lock.
Business sponsors often ask about the usage of an application. They want to know the average number of users, among other usage statistics. Therefore, consider creating programs to keep track of application usage. Usage information can also be used to charge-back to the appropriate corporate departments or divisions.
The last domain category in this list is agreement utility programs. This category is very specialized and does not contain any utility program types. Instead, this category includes all those programs you must create to measure, enforce, and report how well the application satisfies contractual service-level agreements (SLA).
For example, if the SLA specifies that the application will have an availability of 99 percent, then you need to create a program that can capture the application's uptime statistics and compute the availability.
Again, some products, such as IBM Tivoli Monitoring, can measure and report on some SLA attributes, such as availability. You might have to supplement these products with custom programs to measure and report on the other SLA requirements.
Therefore, it is important during the inception and elaboration phases that the development team understands the business's SLA requirements. Then, the team can determine which SLA requirements to measure with custom developed utility programs or with purchased software.
This month I have presented a checklist of five categories of utility programs that support application installation and operations. This checklist isn't exhaustive but it is a good starting point to think about operations support.
Developers usually write utility programs under duress -- that is, on an urgent basis to diagnose a problem, to do research, or to answer operational questions. The challenge is to create these programs earlier in the development process, and save yourself the stress of a last-minute scramble. Awareness of the types of utility programs and how they are used is a starting point. With this information, you and your development team can begin to include utility programs in your project budget and development plan.
Learn
- Quality busters: Who's monitoring the application, anyway? (Michael Russell, developerWorks, September 2004): Find more about the importance of monitoring in a production environment.
- Rule-based access control (Barry Brachman, developerWorks, December 2006): Consider re-using an authorization framework for security enforcement.
- Securing Web Services with Tivoli Access Manager (Chris Kirby, James Hurley, and Roland Lampertz; developerWorks, September 2005): See real-world development problems resolved by IBM's Tivoli Access Manager.
- An SOA Practices Checklist for Building Implementation Roadmaps (Nitin Gandhi, The SOA Magazine, August 2006): Review a related checklist for SOA functionality.
- Wikipedia: utility software: Read a brief discussion of utility programs on Wikipedia.
- Information Systems Audit and Control Association (ISACA): Visit a portal to utility programs developed to satisfy information technology audits.
- US Government General Accounting Office (GAO): Most government agencies, like this one, establish auditing and governance guidelines.
- The Web development zone: Expand your site development skills with articles and tutorials that specialize in Web technologies such as Web 2.0, Ajax, wikis, PHP, mashups, and other Web projects.
- The technology bookstore: Browse for books on these and other technical topics.
- developerWorks technical events and Webcasts: Stay current in your field with these technology sessions.
Get products and technologies
- Downloads and resources: Find downloads and other resources related to Web development.
- IBM product evaluation versions: Get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
Discuss
- developerWorks blogs: Get involved in the developerWorks community.
- developerWorks forums: Participate in any of several Web-centered forums.
Michael Russell has a bachelor's degree in physics and a master's degree in computer science. He was a logistics engineer, a technical services manager, and a certified IT architect at IBM for nearly 14 years and is currently a design engineer in the defense aerospace industry in Orlando, Florida. He has experience in Windows, UNIX, and OS/400 environments and uses Web technology for entertainment through his own company, Vicki Fox Productions.
Table of contents
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags