One of the toughest issues with today's complex software is actually ensuring you are tackling the right performance problems for your application. It's not that tough to run the application, find a hot spot (or two), make some changes, and get a performance improvement. The harder question is whether or not the performance improvement is the right one to have focused on.
That's pretty vague - here's a particular example. Let's say you are developing an editor. There are lots of different operations and functions that someone could do: open files, write text, save files, format, and so forth. People using the editor will have different expectations (requirements) for different functions. Writing text needs to be pretty zippy - faster than human response time and such that a fast typist can't get ahead of the editor. Saving a file will be expected to take a bit longer, but taking minutes isn't going to be very popular. So imagine that someone works on improving the performance of the file save operation from .1s to .05s for a typical 2K file? That's fine, but if that was done instead of making the writing text operation fast enough to keep up with someone entering text, the wrong tradeoff was made.
The editor example was reasonably straight forward (although we can all point to editors even today that don't seem to be very zippy). It can get a lot more complicated when you have a large enterprise application that performs many different operations, all of which have different performance expectations, with different customers using different features. It's even messier if you have to trade off making some operations slower to make a particular operation faster (in the editor case, maybe you make a save operation slower because of the way you've implemented text type-ahead. Maybe that's ok, maybe not...) The net is: you need to understand how your customers use your product to ensure that the improvements you are putting in are important and of value. No rocket science here, nevertheless, difficult to do in practice.
I'm working on a more technical article on actually doing performance analysis - hopefully I will get that out real soon now...