Resolving The Problem

Collecting MustGather data early, even before you open a trouble ticket, helps IBM® Support quickly determine the root cause of a problem. With this information, Support can determine whether:

1. The problem is a known problem (rediscovery).
2. It is a non-defect problem that can be resolved without a product code change.
3. It is a known defect for which a workaround exists.
4. It is a new problem and a product software fix might be required.

Review the information in MustGather: Read first for FileNet Content Engine before contacting Support and before continuing with the recommendations in this document.

Performance MustGather Information

Performance problems occur under various conditions. Be sure to include the following information in the description of the problem:

1. Does the server take a long time to respond or simply hang?
2. Do server system vitals such as CPU and memory consumption display "ramp up" behavior where their usage is increasing over time?
3. Are CPU, Memory, Network, Disk, and other critical system vitals saturated when the problem occurs or are ample resources available?
4. Does the problem occur consistently, intermittently, or only at certain times of the day?
5. Does the problem occur when other background or batch processes are running?
6. Does the Performance problem happen when only a single user is using the server or when the server is under a load?

When the performance problem occurs, collect as much information as possible.  At a minimum, include the following information:

1. CPU, memory, disk, and network usage patterns for the client, Content Platform Engine, and database servers.

   You can use P8 System Manager Dashboard, Perfmon (Windows), vmstat/iostat/sar (UNIX), or tools of your choice to collect this data.

2. Note any abnormal factors and usage spikes such as OS virtual memory swapping, increasing number of threads, increasing number of handles, change in workload.
3. Make sure your server is properly tuned per the P8 Performance Tuning information in the IBM Documentation. Include information such as Java heap settings, application server custom properties, web container thread counts, data source connection pool settings, application server settings, operating system settings, P8, and database settings that are changed from their defaults.
4. Collect detailed Java garbage collection information for the Java virtual machine (JVM). Follow instructions for your specific application server. For WebLogic, add "-verboseGC" in the JVM arguments. For WebSphere, verboseGC is turned on by clicking the check box with the name "Verbose garbage collection", the option is usually located under Application server > server name > Process Definition > Java virtual machine. See the following technote for more details:

   Enabling verbose garbage collection (verboseGC) in WebSphere Application Server.

5. If the Content Platform Engine server appears to hang, create a JVM thread dump. Create the dump by sending a "QUIT" signal to the JVM in UNIX or by using "ctrl-break" on Windows. The WebSphere documentation contains additional information on how to collect thread and heap dumps.
6. If the problem is reproducible with a light load, enable Content Platform Engine trace logging. Trace logging can significantly impact performance, so enable only the traces you need. For instance, start by gathering an EJB trace. You can also enable the Database, Security, or any other trace options you think might be related to your problem. For instance, if you are experiencing full text indexing or search problems, enable the content-based retrieval (CBR) trace. See Enabling FileNet Content Engine Server tracing for detailed instructions on how to enable these traces.
7. When performance is slow under heavy load, turn off all Content Platform Engine tracing in ACCE and remove any overrides to logging JVM arguments.  Then, enable the Content Platform Engine Perflog. Perflog can be dynamically enabled without having to stop and restart the Application Server. (Setting the option "use.me=false" does not disable perflog. It instructs Content Platform Engine to not use the perflog_configure.properties file. To disable writing to the perf.log file, use the option "perflog.interval=off" instead.)
   1. Go to the directory where the Content Engine error log is located.
   2. Edit the file called perflog_config.properties as follows:
      • Set the "perflog.dir=" to an existing directory. The default location is the same directory as the Content Engine error log.
      • Set "use.me=true".
      • Set "perflog.interval=on" to turn on the perflog.
      • Wait a while until you see a file called "perf.log" generated that includes messages.
      • Save the perf.log file and send it to Support for analysis.
   3. Use the perflog utility to create thread dumps and turn on or off verbose garbage collection (verboseGC).

      The Perflog utility also provides a platform-independent way to create thread dumps by editing and saving the perflog_config.properties file, with following line:

      perflog.dumpthreads=<times>:<interval>

      For example, perflog.dumpthreads=10:20 creates thread dumps 10 times and 20 seconds apart. All the thread stacks are written to one file.

      Note the format of the file is not the same as the system-generated file.

      verboseGC can also be turned on and off dynamically using the perflog utility without having to restart the application server. Edit the following line as needed:

      perflog.verbosegc=true/false

      Using the perflog.verbosegc flag is useful when production servers cannot be restarted easily.

   4. Collect any other information that might be relevant. For example, if you have a search or query problem, collect profiles from your database. In SQL Server, use the SQL Profiler with the SHOW PLAN STATISTICS and SHOW PLAN ALL options enabled. You can use TOP SQL or Oracle Stats Pack in Oracle, and the Health Monitor or dynamic SQL snapshots in DB2.