The challenges of introducing new technology
I have spent a significant amount of my time over the last several years on a series of projects across multiple industries in locations all over the world. The most important underlying theme during this time was (and still is) the introduction and promotion of the Service Oriented Architecture concept as a means of organizing functionality in a decoupled, dynamic, and business-aligned manner.
For many organizations, this new concept can be rather disruptive in that it changes the way solutions are designed, implemented, and operated. Companies have to deal with new products and new patterns of solution design, new requirements towards the maintenance and operation of business solutions, and new opportunities for directly supporting the business needs in IT. However, most organizations try to address these challenges with their existing roles, responsibilities, and processes. In some cases, they realize too late that a more fundamental change is needed: a change of processes, organizations, and, yes, culture.
In this article, I want to describe a common issue that I have come to identify as "the challenge of introducing new technology" and look at the best way an organization can deal with this challenge.
Why new technology?
This is a bit of a rhetorical question. Dealing with and leveraging new technology is a part of our lives in IT. In fact, the pace with which new technologies emerge is steadily increasing, and so companies that can leverage new technology quicker than others gain a competitive advantage and are able to deliver real business value faster.
In the context of this discussion, "new technology" includes:
- New software, specifically new commercial, off-the-shelf software products or new middleware products.
- New systems, specifically new hardware platforms or new operating systems.
- New major versions of existing software or hardware.
- New significant functionality leveraged in an existing hardware or software product.
Some of these new technology types are relevant simply because of time and related legal or contractual obligations; new versions of hardware or software must be introduced because vendor support might otherwise be dropped. New generations of hardware are interesting because of improved technical characteristics (for example, faster processors, less energy consumption, and so on).
But sometimes new technology is triggered by emerging IT trends in general. For example:
- The advent of service orientation is an example of such a trend, as something that went from being brand new and not very well documented or supported in actual products, to being today what I consider the state of the art of software architecture and development. IT organizations have had to react to this trend, with varying speeds of adoption.
- Another trend (which happens to be something I deal with a lot) is business process management (BPM). As the name indicates, BPM calls for better management of processes -- for example, a higher degree of automation, shorter cycles, better monitoring of current events -- all of which are only possible if new technology is used that directly supports these goals.
- A possible future trend that might lead to equally large numbers of new technologies and products is the concept of cloud computing, just to name one last example.
Another trigger for new technology can come from the lines of business in an enterprise. New business requirements might arise that result in the need for technology that an IT organization currently does not support. Examples for the technologies that have to be introduced to the IT landscape in those cases are portals, multi-channel architectures, or business rule systems.
A lifecycle for new technology
Despite the wide variety of new technologies (as defined above) and just as many possible triggers for these technologies, you can still define a common lifecycle that all technologies share. The lifecycle described in Table 1 is such an example; there are many variations but all would include these same major aspects:
Table 1. Sample technology lifecyle
|Introduced||A technology or product is brought into an organization for the first time. This is when various groups get to take a first look at the technology, often testing its use in the form of a proof of concept.|
|Planned||Initial planning has been conducted for the technology to go into production in support of a business solution, which will act as a pilot. The planning activities include operational aspects.|
|Deployed||The technology is in production, in a limited fashion, supporting only one or two business solutions.|
|Mature||The technology has reached a high degree of maturity. This includes organizational capabilities, refined processes for both development and operational support. Known best practices have been applied.|
|Retired||No new solutions can be deployed on top of the technology. Existing solutions are migrated over time, if possible.|
Introducing a new technology
A suitable process must exist to support the lifecycle described above. Many companies struggle with this because there is usually no defined process to perform for handling new technology.
In reality, the effort of bringing a new technology into production is mostly tied to the regular development and deployment lifecycle (that is, the activities that are performed for any new business solution) and paired with the individual heroics of experienced staff.
In particular, it has been my experience that operational characteristics are considered late -- or not at all. These characteristics include things like monitoring, capacity planning, problem determination, and backup/restore procedures, none of which deal with creating a new solution, but instead deal with operating and maintaining it. The key point to remember is that a new technology will bring new requirements with it that didn’t exist before.
Thus, I strongly recommend establishing a process that formally defines steps that are required whenever a new technology is introduced to a company’s IT landscape. Within such a process, you can ensure that the operational aspects are executed -- and executed as early as possible.
Describing such a process in detail is beyond the scope of this article, but it will be helpful to look at a sample list of essential considerations that are most often overlooked:
- Technology testing
As mentioned earlier, the introduction of new technology is often embedded in the process of developing and deploying a business application, and this includes all testing steps. And while these tests include non-functional and operational testing, there are no steps specific to the fact that new technology is being introduced.
For example, new ways of monitoring (plus appropriate alert and report definitions) might have to be established, possibly along with using new tools for monitoring. Or, new operational procedures might be needed to ensure high availability. Or, performance tests have to be run to determine initial benchmarks.
All of these aspects have to be thoroughly tested before the first production rollout. Actually, this type of test can happen in parallel to the actual development process because the relevant test cases do not require that the business application already exists. The test exclusively focuses on operational aspects that are independent of any particular business application.
- Operational runbooks
An operational runbook is basically a document (or a set of documents) describing how to operate a system. It captures procedures and best practices, gives guidance and prescriptive information, and serves the needs of operators and support personnel.
Runbooks exist in almost all IT organizations, but their creation is rarely standardized or included as part of the normal process -- which is why it’s on this list. Operational staff should be involved in the introduction of new technology as early as possible, and the formal creation of runbooks is a good way of getting an operations team acquainted with technology they have never before operated.
Every IT organization has an approach toward funding their activities. A business-sponsored project will have a budget based on a business case that balances cost against the benefit of a solution. IT teams have defined ways of charging for their services.
While this is business as usual, the arrival of new technology brings extra considerations with it that have to be dealt with. For example, the new technology introduction process (described above) needs to be funded; the first business pilot project using a new technology should not be burdened with the extra cost of introducing it. Instead, the cost should be spread according to expected usage per the technology roadmap.
Moreover, this factor speaks yet again to the necessity of getting operational teams involved early so that they can better estimate the cost of operating a new technology.
It’s all about governance
Assume for a second that a technology lifecycle and a formal process for introducing new technology exist. What’s the next challenge? Any process is only as good as its enforcement mechanisms, compliance criteria, and exception and escalation procedures. In short, you need proper governance.
Being able to enforce the execution of a defined process is a challenge in any environment. If no mechanisms exist to prevent process steps from being ignored, then ignored they will be, because of time, budget, or other business pressures.
Defining proper governance should therefore be a core part of defining and implementing a new technology introduction process. As with all aspects of governance, sponsorship from senior executives goes a long way. There should be formal compliance criteria and related checklists that gate the progression from one part of the process to the next. Relevant reviews and audits must be embedded in the process itself, together with appropriate role definitions (for example, separation between the new technology owner and the new technology reviewer).
The lack of a formal process for introducing new technology into an IT environment is one of the biggest challenges faced by companies looking to leverage new products. Such a process guides a technology along its lifecycle, ensures proper and timely involvement by technical teams (like operations), and defines proper governance mechanisms ensuring the process is always properly executed.