IBM Support

Why collecting the Websphere "MustGather" data for opening a performance PMR really is beneficial

Technical Blog Post


Why collecting the Websphere "MustGather" data for opening a performance PMR really is beneficial


So, you think you have a performance issue that requires opening a PMR with IBM Maximo support. You open the PMR, and the first thing that comes back is, "Please provide the MustGather information".
We're all in a hurry these days, and it seems like such a hassle to provide this info. However, read on, and I'll explain why the MustGather data for your Application and HTTP Server(s) can provide invaluable insight into your environment for solving performance issues.

Just to recap, the data you are asked to collect for your Application and HTTP Server(s) consists of the following information:
  • Application Server Hardware Configuration
    • Operating System (Name, Version, 32-bit / 64-bit)
    • Websphere Application Server Version (32-bit / 64-bit)
    • Hardware Manufacturer/Model
    • # of CPU’s
    • CPU Speed
    • Physical Memory
    • Disk Size
    • Is this server an LPAR? If so,
      • How many LPARs per physical server?
      • How much physical memory is allocated?
      • How many physical processors are allocated?
  • Load Balancing Configuration
    • Do you use hardware or software load balancing? (Explain)
    • Is IBM HTTP Server used?
    • If so, what is the load balancing policy (weight based, round-robin, random)?
  •  Authentication Configuration
    • Do you use Single Sign-on?
    • Do you use LDAP synchronization?
  •  Cluster Architecture
    • Do you have separate clusters for:
      • User Interface (UI) processing?
      • Cron Tasks?
      • Integrations?
      • Schedules / Report Jobs?
    • For each cluster implemented, please provide the following data:
      • Are there other applications deployed to the same server?
      • How many load balancers do you have?
      • How many servers?
      • How many clusters?
      • How many members in each cluster?
      • How many JVM’s per Server?
      • Memory allocation per JVM (minimum and maximum heap size)?
  •  WAS and IHS Plug-in Configuration
    • Please send the output from the following command:
      <WAS_HOME>/java/jre/bin/java –fullversion
    • For each cluster member, please send the following file:
    • For each cluster, please send the following file:
      <WAS_HOME>/profiles/<profilename>/config/cells/<cell name>/clusters/<clustername>/cluster.xml
    • For each IHS server, please send the following file:
  •  Websphere Logs
    • Please send copies of the following logs:
      SystemOut.log, SystemErr.log, native_std_err.log
 So, now that we've taken the time to gather this information, how can it be used to resolve potential performance issues related to WAS / IHS?
  •  Server Hardware Configuration
Different operating systems can be tuned differently, so it's helpful to know whether the application server is running on AIX, Windows, Linux, etc. Knowing if the OS and WAS are both 64-bit determines whether or not the JVM heap size can be larger than what's available on 32-bit operating systems. And, some folks prefer to run the 32-bit version of WAS even if the OS is 64-bit. Also, understanding the version of WAS / IHS installed can help rule out known issues with back-level releases of the middle-ware. Similarly, knowing the CPU, memory, and disk info can indicate whether or not the hardware has the capability to handle the application server load. Lastly, there are special considerations that must be taken into account when the environment is virtualized.
  •  Load Balancing Configuration
Many performance issues can be traced to the system not properly load balancing across JVMs or servers. There are many ways to do load balancing, so understanding whether hardware or software is used can be important. Likewise, knowing if IBM HTTP Server with it's WAS plug-in is being used and what policy for load balancing has been chosen can be important factors in diagnosing load balancing issues.
  •  Authentication Configuration
Bottlenecks can occur at any point in the solution, so knowing if single sign-on or LDAP group synchronization is used can be important to rule out potential issues in those areas.
  •  Cluster Architecture
The Best Practices for System Performance white paper highly recommends that you setup multiple clusters to handle the various types of processing that come into the system. The main goal is to ensure that the JVMs servicing the user interface requests aren't being bogged down by log running escalations or reports. Similarly, you want integration processes to be free from the UI when performing their transactions. Knowing the cluster architecture decisions that have been made can help pinpoint issues that may be related to various types of processes conflicting with one another. Once the overall architecture is understood, it is important to know how each cluster is configured, the number of servers involved in each cluster, the number of JVMs used, etc.
  •  WAS and IHS Configuration
 Knowing the exact level of java installed in Websphere can pinpoint known issues in back-level versions of Java. While applying WAS fixpacks will normally bring the version of java up to date, the WAS-SDK fixpacks much be installed separately from the WAS, IHS, and Plug-in fixpacks. It can be easy to overlook the WAS-SDK fixpack, resulting in back-level versions of java which can have memory management and other issues. While all of the configuration information for WAS and IHS can be easily obtained by looking at the WAS Administration Console, that level of access isn't available when diagnosing a potential issue remotely. However, obtaining copies of the server.xml, cluster.xml, and plugin-cfg.xml files provide the same level of information as if using the WAS Administration Console.

By reviewing the server.xml files, you can see the JVM minimum and maximum heap sizes, generic JVM arguments (such as the garbage collection policy in use), and thread pool settings for each configured JVM, These values can then be compared to the system capabilities and the performance best practices. The cluster.xml file shows the information about the cluster members assigned to that cluster. Lastly. the plugin-cfg.xml file shows how the IHS WAS plug-in is configured to load balance across the JVMs, including the load balancing policy in use.
  •  Websphere Logs
Analyzing the SystemOut.log and SystemErr.log can uncover many issues at the application server level that could affect performance. With verbose garbage collection enabled, the native_std_err.log shows any potential garbage collection issues that can have a negative effect on performance.
As you can see, all of the requested "MustGather" information can be used to understand the architecture, environment, and configuration to diagnose potential performance issues. So, next time you suspect you have an issue related to WAS / IHS, know that the time taken to collect this information is well worth the effort!

[{"Business Unit":{"code":"BU005","label":"IoT"}, "Product":{"code":"SSLKT6","label":"Maximo Asset Management"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":""}]