Born to perform?

Having worked as an application developer, I learned fairly quickly that I had to troubleshoot and debug my problems and figure out what needed to be fixed if I was going to be successful. Thankfully, I was rather successful as a developer because I spent a lot of time debugging my mistakes -- and I made many, but that’s a topic for a different article.

When I became a performance specialist, my background served me well, since a successful performance specialist requires the ability to learn from past mistakes (mostly the mistakes of others). Now, as a senior member of the performance team, one of my responsibilities is to mentor and train other IT specialists all over the world to become performance specialists. Why would you want to be (or need to have) a performance specialist? Because a performance specialist can help an enterprise in a few critical ways:

• Applications that have poor performance can also require more resources (such as CPU, memory, disk, and so on), which means higher operating costs. A performance specialist can tune an application and the application environment, possibly resulting in substantial operations cost savings. On one performance tuning exercise I was involved in, we were able to extend the expected increase in hardware expenditure by a few years simply by tuning the environment. Tuning the operational environment, however, is not where an enterprise gets the most bang for the buck. Tuning the application is where the biggest gains can be made.

• Applications are often mired with well intentioned code that ultimately does not provide desirable performance characteristics. Slow or poorly responsive applications can actually turn users away, especially if you suffer production outages. Conversely, applications that return answers promptly and efficiently often lead to more sales and customer loyalty. Having a performance specialist work with your development team can yield desirable and impressive results. On more than a few occasions, I have worked on engagements where fixing code-based performance problems enabled the operations environment to shrink by factors of 10. Just think of the extra test environments you could build with those kinds of savings!

• Performance specialists like to tinker. A performance specialist will not only run extensive tests on your test environments, but also run negative tests to see how an application will respond to negative events (for example, if the network to the backend goes down). It is this kind of curiosity that a performance specialist bends and molds to get to the root cause of many potential problems -- before they happen in production.

So, what exactly does a performance specialist do to help an enterprise achieve such substantial cost savings? A few different skill sets enable this person to analyze data and decide what steps to take:

• A performance specialist typically has a background in run time operations, either as an administrator or in some other supporting role, plus a variety of skills related to modifying parameters in the operating system, network, application server, databases, and so on. This doesn’t mean to imply the performance specialist will have detailed knowledge about every hardware and infrastructure component there is. I know I frequently rely on the subject matter expert who administers or supports that particular network element for answering questions or providing details related to the problem. But I am well versed enough in the technology to be aware of whether that particular piece of the infrastructure is participating in a negative event or not.

• Performance specialists must have a background in application development. Since application code is the primary reason for performance-related problems, it takes someone with a development background to understand not only how to identify what is wrong but also how to fix it. This means that the performance specialist also either has a good troubleshooting background or has access to someone that is a good troubleshooter.

• Testing is key to performance activities. In fact, testing is probably where a majority of the performance specialist’s time is spent. We run test after test, collecting data from each test and determining whether we see an improvement or degradation. Use cases have to be discovered and test plans based on the use cases have to be developed. A test team then executes the test based on the test plan and the performance specialist analyzes the data. Started early in the application development lifecycle, this one activity can provide the most noticeable results, particularly if the performance specialist has a development background.

• Performance is not always a technology problem. Sometimes there are organizational or human processes that need minor or major refinement. For example, if operational processes break down and a hot fix is applied to some servers and not others, then a change was approved but not executed properly. Sometimes it means educating administrators on how to make changes or educating change managers to detail change records clearly. On other occasions, it could be fundamental problems in the organization that prevents the enterprise from operating optimally. The performance specialist must be able to recognize when a problem needs a non-technical solution and bring those issues up to appropriate management for action. In my experience, the most common, non-technical problem is that an organization’s staff restrictions result in people filling multiple roles. It might look good on paper, but an overwhelmed staff will make mistakes, which can be expensive if production outages occur as a result.

Now that you can begin to understand the value of performance specialists, how can you become one (or grow one)? Listed below are the steps I take my IBM performance specialist protégées through. This will give you a glimpse of the makings of a future superstar.

1. Get a mentor. This is how I got my start as a performance specialist. First was Dr. Robert B. K. Dewar who taught me the basics of troubleshooting back at the Courant Institute of Mathematics at New York University. I had a fairly perplexing problem in my application (written in 8088 assembly, no less) which I couldn’t figure out. I kept coming up with different solutions to try and fix my problem, but I could not get any of them to work. Dr. Dewar gave me basic training in identifying the problem. Once I knew that, figuring out the solution was a lot easier. That basic troubleshooting training has been the crux of my career, and it’s the basic training I start my protégées on. There is only one useful way to learn this skill and that is by actually fighting fires, so I will typically have someone shadow me when I work on a problem. I show how to identify that a problem has occurred, and go through the typical steps to solve that particular problem.

   Most performance specialists I have worked with usually have a strong development background. The actual technology (say, Java™ or C++) they are experienced in is less important, because at the end of the day every language is reduced to the same bits and bytes that the computer understands. Understanding what a stack is and how to read and traverse it has been one of my fundamental skills for the past 20 years, regardless of the language or platform.

   A protégée does best with multiple mentors; this is true for any discipline. This is because each person has his or her own set of tools and approaches for solving a problem. One particular approach is not necessarily better than another, but knowing different techniques and being able to think outside the box are valuable skills. You can often learn new tricks by seeing how someone else solves a problem.

2. Listen to the machine. The computer never lies. If the machine is experiencing a crushing memory headache, it says so. The key is in understanding what the machine is reporting. If someone misinterprets data, an incorrect conclusion will not lead to a solution. To be successful, a performance specialist requires the ability to not only collect the data but also to correctly analyze it.

   And this is where tooling comes into play.

3. Have an arsenal. A performance specialist is only as good as the tools he uses. There are a lot of tools out there, both freely available and commercial. Each offers another way to examine data and provides insight to the problem. Some tools provide analysis capabilities that might be single-sided (that is, they don’t see all the facets of a problem), so they might offer suggestions that work only within their sphere and not within the scope of the system as a whole. Having the right tools and knowing how to use them is crucial for a performance specialist to be successful.

4. Be diversly trained. If you didn’t catch any of the subtle hints above, then here it is in black and white: get cross trained. Performance specialists are jack-of-all-trades people. With a background in development, they will likely resist the temptation to rewrite an application. With a background in managing and running a production environment, they will understand the stress and strain people in these roles can have. With a background in various elements of the infrastructure, they can make reasonable assumptions on whether a particular element is participating in the problem or not.

   Frankly, it’s a good experience for application developers to be rotated into the operations and administration role for 3-6 months, even if they are not targeting to be performance specialists. The experience of seeing how the other side lives can only be achieved by walking in their shoes and typing on their keyboards.

5. Add bricks to your wall. Education is a key directive in IBM® for growing performance specialists. I helped develop course SW612, which is a one week performance-oriented boot camp. Clearly, someone is not going to attend the boot camp and come out a performance expert, but the education does cover the basic training you need to get started. We cover topics such as how to produce and read a javacore, analyze memory problems, and how to use some of the freely available tools from IBM alphaWorks and developerWorks. One tool I recommend every performance specialist gets ahold of is the IBM Support Assistant, a tool that is the result of hundreds of IBMers that perform troubleshooting for a living, that has some really neat troubleshooting and analysis capabilities.

Although not everyone will venture to become a performance specialist, some of the suggestions presented here might be good ones for you or your organization to follow in general: having a mentor is important for anyone, moving people around to fill different roles (even if for short periods of time) is good for their own personal growth and strengthens the environment, and education is key to anyone’s growth regardless of their role or subject matter. But if you are (or have) a good candidate for a performance specialist, then the steps presented in this article is a good way to start.

Resources

• Book: How to Break Software: A Practical Guide to Testing, James A. Whittaker, Addison Wesley, 2002
  
  
• Alistair Cockburn: Resources for writing effective use cases
  
  
• IBM Support Assistant
  
  
• IBM developerWorks WebSphere zone
  
  
• IBM alphaWorks
  
  
• IBM developerWorks
  
  

About the author

Alexandre Polozoff is a Senior Certified IT Specialist in the IBM Software Services for WebSphere Performance Technology Practice for the WebSphere suite of products. In this role, he works with IBM customers on various high volume and performance related engagements. Mr. Polozoff has an extensive 20 year background in network and telecommunications management, application development, and troubleshooting. He has also published papers and speaks at various conferences on performance engineering best practices.

Rate this article

Comments

Back to top

Table of contents

• Born to perform?
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into WebSphere on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers