Everything runs slowly at unpredictable times

The best tool for this situation is an overload detector, such as the filtd daemon, a component of PTX.

The filtd daemon can be set up to execute shell scripts or collect specific information when a particular condition is detected. You can construct a similar, but more specialized, mechanism using shell scripts containing the vmstat, iostat, netstat, sar, and ps commands.

If the problem is local to a single system in a distributed environment, there is probably a pathological program at work, or perhaps two that intersect randomly.