Level: Intermediate Michael J. Welsh (writer@roninwriter.com), Senior Writer, Ronin Writer
02 Oct 2007 After successfully building your new IT enterprise architecture, it's time to test
it. Testing proves that the hard work you and your team have put in really works. By stressing the new
architecture, you'll know where its weaknesses are and how well it will serve the
enterprise.
Now that you've built your enterprise architecture, you must test it. New technology,
especially technology that's on the bleeding edge, will have some kinks to work out.
Testing the information technology (IT) architecture is all-inclusive. Testing should
include the hardware, applications, and the people responsible for those systems.
Testing is a great way to break things and get paid for it. While no one ever wants
their IT systems to break, odds are that they will. By testing common failures, you
can help justify the additional costs of redundancy. Throughout the testing process,
communication is essential. Giving test users a heads-up about possible problems and
what they should do when those problems occur helps instill confidence in the project
and its members.
Preparation
When preparing to test your architecture, you must first decide what to test. Simple
changes, such as swapping CRT monitors for flat-panel displays, obviously don't
require rigorous testing. But the new fiber-optic backbone should be load tested
with new applications and hardware. Meet with the IT team to decide how time should
best be spent testing.
Testing has three functions: It allows you to determine how new applications will
perform in a production environment, it verifies how well redundancy will work in a
failure state, and it helps determine how well your team members react to problems.
Testing applications can be a rather straightforward but intense process.
Applications can have hundreds of little-used functions that you must test. For
example, accounting applications have some tax-related tasks that are performed at
the end of the month and other tax-related tasks that are performed only at the end
of the year. It is best to have these functions working properly before the tax man
comes around.
Skills and competencies
Before testing, look over the whole enterprise forward, backwards, and from every
angle. Consider its strengths and weaknesses. Through a thorough evaluation of the
entire enterprise, you will know which systems need the most comprehensive testing.
If at all possible, testing should involve non-production systems. If testing must
intersect with production systems, explain to managers the need for the testing and
why it affects a live system, and schedule the appropriate downtime. Also, be sure
that a good backup exists for any live system to be tested. Verify this by restoring
to a virtual server and comparing the data sets with the live unit.
Virtual environments
Simplify the creation of test environments by using virtual systems. Virtual
systems are easy to set up, break, and restore. Begin by creating one standard
server and cloning it to make the other servers. After any applications are
installed, many virtual PC programs allow you to take a snapshot of current
settings to facilitate restoration following a failure. The benefit is that
you can create a new server in minutes rather then hours.
VMware offers a wide selection of choices for creating a custom virtual
environment. You can choose from dedicated virtual machine (VM) software that
requires no base operating system, and you can run several servers at once.
The biggest advantages to using VMware are its ability to work with multiple
operating systems, its easy configuration, and its testing tools.
Several network simulators are also available to help test new configurations
and assess failover scenarios. While most are designed for preparing for
certification testing, they can offer a great benefit for building a virtual
server environment at a reduced cost relative to buying the real hardware.
Checklists and staff preparation
With the new enterprise architecture ready for launch, build a checklist of core
systems, and list those backup systems that require testing. Core systems will
be apparent from the documentation the team kept. Core systems will be the
hardware and software without which the organization cannot exist. Often, such
systems consist of the critical servers, the network backbone, and messaging
pieces such as e-mail and telephony, as shown in Figure 1.
Figure 1. Core systems
This diagram doesn't show the details found in a complete network diagram;
rather, it's meant to highlight the pieces of the enterprise that are the most
crucial to business operations.
Preparing support personnel and users for those questions and issues most likely
to arise with the new system helps reduce help desk calls. Your team members
should assemble a list of problems they're already aware of, such as an
application hanging, and build a list of known fixes.
Deciding when testing can be done is one of the more difficult chores. Finding
a time when everyone is available during the day can be tricky, so after hours
may be a good time. However, you may encounter resistance from team members not
wanting to spend more time on the job or a boss who doesn't want to pay for
overtime. The best balance is deciding on the staff needed to run the test and
the best time to do it. Other team members may be designated to an "on-call"
function and brought in only if needed.
At this point, you have already selected compliance standards for your new
architecture. Some have specific points of failures that must be tested before
being launched into a production situation. Items such as firewalls or security
application components require verification before supporting live data.
Build specific checklists of what should be tested and how to conduct that
testing. Additional checklist content may include:
- Screen captures of what the data should look like before and after
testing.
- Separate listings of specific tasks.
- Checklists divided by the groups that perform the tasks. Server
hardware build may be a different group from the people who load and
configure the applications.
Document procedures
Testing builds documentation. It applies the struggle of daily life to the new
IT architecture.
At a minimum, you should have three types of documentation: historical, procedures,
and disaster recovery. The first type, historical documentation, should
include the details of how everything in the architecture is built. This document
should contain diagrams of work flows and notes on how the pieces are connected. As
the architecture evolves, minor details can be forgotten—for example,
a document explaining that the payroll software connects to the warehouse database,
because order pickers get bonuses for accuracy (see Figure 2).
Because this setup is transparent to the warehouse database, the warehouse database
administrator (DBA) must be aware of the connection before making changes.
Figure 2. Warehouse wireless diagram
Procedural documentation consists of be checklists of daily, weekly, or monthly
tasks. Examples include daily tape backup change procedures, checking logs, or
reporting problems. Some systems may require routine maintenance to continue
running efficiently. Daily procedures are generally given to specific people or
groups responsible for those areas. Having detailed steps of the tasks helps
ensure that the tasks are done correctly and on time but also helps those staff
members filling in for other people. Procedures should include common errors and
how to correct them as well as standard setup procedures for introducing new
hardware and applications to the enterprise. As this task will pass through
several groups, you must have documentation specific to each group. A document
flowchart such as the one shown in Figure 3 helps list
prerequisites and what each group is responsible for.
Figure 3. Server setup documentation flow
For uncommon errors, you must have disaster recovery documentation. This specialized
documentation is a combination of historical documentation and daily procedures.
While you cannot plan for every disaster, many of the recovery procedures overlap.
Documentation procedures for simple tasks, such as starting a server, can seem
trivial at first but can be a source of confidence when testing a disaster scenario.
When the unexpected happens, it's not uncommon for panic to set in quickly among
the support staff. Tensions rise at the help desk as calls come flooding in about
a server-based application that has stopped responding. Good documentation,
including common sense details, help, because people working to resolve the issue
may forget that a server restart will affect other systems.
Documentation procedures for disasters should also include whom should be informed
when a disaster begins and when it's resolved. Such staff members may include shift
managers, department heads, and even the big chief of the company.
A lot has been written on disaster recovery preparation. See Resources
for links to more information.
Auditing
Documentation is a living and evolving entity of any enterprise. As your IT structure
grows and changes, so must your documentation and testing procedures. When your
documentation changes, you must test it to make sure that it remains accurate.
Say that a year from now, your company upgrades its e-mail software. This new
software has a slightly different method for restoring mailboxes than the previous
version. If the changes aren't updated in the disaster recovery documentation, it
will slow down the engineer performing the recovery. Not only will this make for an
unhappy engineer, but people waiting for the e-mail server may get upset at the
length of time it takes to restore functionality.
By scheduling regular testing of the architecture components throughout the year,
you can update and polish the documents that you have already created. These
periodic tests also build confidence in the support staff. So, even if an incident
doesn't go by the book (and most don't), team members will have the experience
from testing to make better decisions under pressure.
Auditing your testing results can have a large impact on the confidence of the
business as a whole. Most testing results are only viewed by senior managers,
who are probably the same people who control the money. Often, they are not
technical people, so explanations of the tests and the results are useful.
The use of change-management software, such as IBM® Rational®
software, to track changes in applications manages application testing. Help desk
applications, such as Numara Track-It,
can generate reports to assess where new problems are emerging. This information
is used to create tests that are more accurate to the needs to the enterprise.
You can create tests to verify that fixes to problems are permanent and that the
same problems don't recur.
Don't worry about tests that produce negative results. Failure is a good thing when
testing, because the failure happened in a controlled environment and not during
production. Use failures as learning opportunities. Why did the system fail? Could
the failure have been avoided? Is there a way to prevent it from happing in
production? Also, don't try to hide failures in audit reports. Use them to
highlight the success that came from it. Point out how the problem was not
foreseen in development or how well the team reacted when the problem surfaced.
Compliance
Testing and its results may be required to meet compliance obligations with certain
standards, such as the Sarbanes-Oxley (SOX) Act, Payment Card Industry (PCI), or
the Health Insurance Portability and Accountability Act (HIPAA). Some of these
standards have a set amount of time that you must retain test results. Depending
on your organization's relationship with its customers, some of these testing
results may be shared.
You may wonder, why bother with compliance at this stage? When you test, you're
testing the compliance standards. The goals of compliance should not be to
reach the minimum requirements but to exceed them.
Compliance testing and auditing by a third party may be more beneficial than
performing them in-house. Third parties will produce a less-biased perspective
on how audits and compliance are viewed. Audits may be performed either externally
or internally. External tests are generally the easiest, as most firms can test
from their location, and there is little setup. Internal compliance audits that
test certain features or events can be rather invasive. A third party will need
to learn the IT landscape and will need to know which components must be tested.
Using a respected auditing firm is going to have more standing when sharing results.
Companies such as Solutionary or
Ambiron Trustwave specialize in specific
industry and compliance auditing and testing. They are an excellent source for
finding which standards and testing you need for you specific industry.
Tools and techniques
Because no two architectures are built alike, neither can the testing techniques be
the same. Building testing metrics for key components takes some time, so consult
with your team. An excellent place to start is with the IT architecture guidelines
published by The Open Group. Similarly,
Tetworks has an open source tool kit
that can test applications.
Some tests may need to have special considerations. If power fails, the uninterruptible
power supply (UPS) will take over. But how do you develop an accurate test? Often,
the answer is with the manufacturer's instructions. In this case, many UPSs have a
self-test feature that will not jeopardize equipment plugged into them.
Some testing may not be as simple as looking at individual components. How would
you verify shifting production traffic to a backup datacenter without actually
disrupting work? Through careful planning and clear communication, you can achieve
such testing, although it will probably be done after hours.
Another consideration is the amount of time testing will take. How much time it will
take to run each test? Will this be on a live system or from a data restore on a
virtual system? If this is a live system, plan time to repair any damage a test may
cause, such as a failed router or hard disk. Several tools are available to help
with your testing. See Resources for more information.
Milestones
Use the following milestones to set objectives and keep your IT enterprise project
moving along:
-
Set the objectives: List what needs to be tested and how frequently.
Set deadlines for documentation to be completed; otherwise, it may not get
done. Especially make certain that documentation is updated after testing
finds errors within the procedures.
-
Management: Manage testing throughout the process. Make sure that
it's done to the established standards, and review the results with team
members.
-
Communications: Communicate with the team and with managers about
why testing is important. Testing often gets a low priority because of
other projects or often gets rushed to get something into production.
-
Maintaining resources: Assess the skill sets team members possess
and have them test their areas of expertise. It's also a good idea to
cross-train other personnel within testing as well as normal operation.
Doing so gives team members an opportunity to react to a failure state
in a controlled situation. Be sure to add testing into your budget. This
may mean simulator software, test equipment, and overtime if you need
to test production equipment after hours. Budget costs may also include
professional training for staff members if they need a jump-start on
operating systems or applications with which they're unfamiliar.
-
Determine whether outside services will be needed: It may make
sense to have an external group perform the audit. Professional auditing
groups can do a thorough job of finding all resources and asking the
right questions. Outside services can be especially helpful if you don't
currently have the proper tools to perform a detailed assessment.
-
Assign roles: Now the rubber meets the road. Hand out tasks that
best suit individual team members, and set them loose.
-
Maintain documentation: If you haven't begun creating documentation
prior to starting this step, it's time to start. Collect operating
procedures, code changes, and help desk procedures. It may be a good time
to have team members document what they really do instead of what they were
hired to do.
 |
Summary
Make testing a critical function within your new architecture's procedures. Good
testing fundamentals bring together the function of your new systems and the
redundancy behind them. Testing your new enterprise architecture illustrates the
hard work and dedication that you and your IT enterprise team members have spent
building it.
Remember, failures that result from testing are good. Learn from them to sharpen
your team's skills, and create new tests to determine whether the error can occur
again. Testing is a battle drill for the enterprise team. Throughout the testing
stage, communication is important. Keep members outside your group informed about
when testing will take place. Not all testing will be done during normal work
hours, so plan and budget ahead for testing time and even dedicated testing
equipment. Keep close records of changes to the architecture, and build test
procedures to keep track of those changes.
Resources Learn
Get products and technologies
-
Download IBM
product evaluation versions and get your hands on application development tools
and middleware products from DB2®, Lotus®, Rational, Tivoli®,
and WebSphere®.
-
Several resources are available to help in your testing endeavors:
Discuss
About the author  | | | Michael Welsh is a 15-year IT professional specializing in IT security, disaster
recovery, and networks. He is also knowledgeable in operating systems, hardware, and
many server-side applications such as Microsoft Exchange Server. Michael writes
technical articles and documentation for Web sites and businesses. |
Rate this page
| 
