IBM Engineering Requirements Management DOORS Next V7.X Performance MustGather

Step 1: First, make sure you followed our recommendations for Version 7.x.

• Database:
  • Oracle database tuning.
  • DB2 database tuning.
  • SQL server: DOORS Next V7.x using SQL Server does not scale to enterprise-sized data so evaluate your data size by using the following information:  Thresholds for determining whether your DOORS Next instance exceeded the capacity of Microsoft® SQL Server.

• DN Server
  • DN Server Tuning.

If the recommendations are not in place, then make these changes and check the performance again. If the issue persists after these settings are in place, then proceed with gathering the data in the next steps.

 Step 2:  Narrow down the scope of the investigation:

• The scope needs to be narrowed to one issue alone. If you reported, for example "DN is unusable" or "Nothing works", this information does not provide enough detail in the case for Support to proceed with an investigation.  We need to see an example of the issue including a detailed description, steps to reproduce, screen prints or videos clips showing the problem.  For example, if you are having performance issues loading module views, then compare the affected views for similarities.  Then, focus on one of the modules' views having the issue and gather the following information.

Step 3:  If not already provided in the case, or included in the additional output within this document, supply the environment details:

• Database version.
• Confirm when the last DBStats was run on the Database.  - Version table runs daily and others weekly.
• Is it a distributed environment?
• Using physical or virtualized environment?
• Is the platform Windows or Linux?
• Using Liberty, or WebSphere (Include Version)?
• The total No. Of CPUs?
• The RAM allocated to RM server?
• The JVM heap for RM server (-Xmx setting in server.startup.bat or server.startup.sh)?
• The Approx.# of projects?
• Is Configuration Management enabled for projects?       
• Approx. # of artifacts per project?

 Java cores supply much of this information.  See: 4. Gather Java Cores

Step 4:  Reproduce the issue and gather the following information:

1.  Active Services page: 

1. Navigate to https://<JazzServer>/rm/admin#action=com.ibm.team.repository.admin.activeServices
2. Take a screen print or save the web page as HTML and upload it to the case for review.

2. Gather Browser Logging (HAR file):

a. Using Firefox:

                                                    i. In Firefox navigate to the "Open menu" (Hamburger icon on the upper right side of the browser)

                                                  ii. Select On the Developer Option.

                                                iii. Select the Network Option, a separate window opens.

                                                iv. Select the clear option.

                                                  v. Perform the steps that result in the performance issue.

                                                vi. From the Network window, you must right-click and select "Save all As HAR".

                                               vii. Upload the file to the Case for review.

b. Using Chrome:

                                                    i. To enable Developer tools, select the options button at the end of the web address bar

                                                  ii. Select "More Tools" followed by "Developer tools".

                                                iii. The Developer tools browser window opens at the end of the screen.

                                                iv. Clear the Network and Console Tabs (press the clear button in the Network and Console tabs).

                                                  v. Perform the required test that you want to gather Browser logs for.

                                                vi. From the console window, copy and paste contents to notepad.

                                               vii. From the Network window, you must right-click and select "Save as HAR with content"

                                             viii. Upload the files as well as the associated rm.log to the Case for review.

3. Gather QUERYSTATS Data:

a. Go to https://<JazzServer>/rm/admin -> Advanced Properties, set "Enable REPODEBUG service" to 'true' and then Save.

b. Open https://<JazzServer>/rm/repodebug/database/queryStats and select "Stop" followed by "Reset" and then click "Start" again.  Leave this tab open.

c. In a separate tab, perform one of the operations that is slow or times out, several times if possible, wait 15 minutes.

d. Back on the QUERYSTATS tab opened in step b, click Stop and then save the entire page as html.

e. Upload the data to the case.

4. Gather Java Cores:  

a. Navigate to https://<JazzServer>/rm/admin#action=com.ibm.rdm.fronting.server.web.logging.

b. Scroll to the end and trigger an “IBM JVM JavaDump” (run this 5 times, 1 minute apart or periodically if the test takes longer).

c. Upload the javacore.YYYYMMDD.HHMMSS.XXXXX.XXXX.txt files to the case for review.  

d. Alternatively use the following process to automate the collection of Java cores: WAIT tool

5. Gather the Verbose GC logs:

a. Review Enabling verbose garbage collection (verboseGC) in WebSphere Application Server

b. For Liberty, it is automatically enabled and the files are located here: “server/liberty/servers/clm/”.

c. If the problem is an Out Of Memory Exception, or large amount of heap being consumed then use: Generating a core dump file and preparation of core file for analysis on Linux systems   

6. Gather a full set of DN ISADC Logs:

a. Review Running IBM Support Assistant Data Collector

b. Upload the resulting ".zip" file to the case.

 7. Check the Configuration cache settings:

a. Run this SQL query against the database: Number of Active (nonarchived) configurations in the Repository:

• For SQL Server & DB2:
  • SELECT COUNT (*) FROM VVCMODEL.CONFIGURATION where type_col in (0,1,2);
• For Oracle: 
  • SELECT COUNT (*) FROM RMUSER.VVCMODEL_CONFIGURATION where type in (0,1,2);

b. If the count is greater than 2,000, follow the steps here: Tuning the configuration cache for IBM Engineering Requirements Management DOORS Next V7.X.

8.Gather the SQL Monitor report and execution plan details:

• We need to understand how the database is executing complex queries, so for any case where you see long-running SQL in https://<JazzServer>/rm/repodebug/activeSql or https://<JazzServer>/rm/repodebug/database/querystats run the following during the execution of a query

a. Oracle:

                                                    i. SQL Monitor report for the long-running statement in SQL Developer, see more details here: MustGather information for diagnosing long-running SQL statements on Oracle

                                                  ii. An execution plan (that includes predicate and column projection details), find the steps required here: Getting detailed execution plans 

                                                iii. AWR report, see: Oracle.com: Comparing Database Performance Over Time

b. DB2:

                                                    i. Request the db2diag.log for review:  Determining the location of db2diag.log for a Server instance 

                                                  ii. db2support is a built-in DB2 tool that collects all-important DB2 configuration information as well as certain system catalogs relevant to the query optimizer. To run the tool, use the following command as Db2 administrator:  db2support -d <dbname> -cl 0 (db2support compresses all information collected in a single ".zip" file).

                                                iii. Using db2mon gather the report while the issue is reproduced.

1. For DB2 running on Linux follow the steps here: Running Db2Mon (Linux). For Linux, there is also an automated process, so instead of manually running db2mon and collecting EXPLAIN PLANs, an alternative way is provided to automate the whole process that automatically gathers the Explain plan. See: Automating the db2mon data gathering

2. For DB2 on Windows follow the steps here: Running Db2Mon (Windows) 

3. Once the report is available, follow the steps here to analyze the data:Analyzing the report output, for any long running queries identified, gather the Explain plan, see: Explain plan 

4. Upload the data to the case for review.

    c. SQL Server

                                                    i. Gather the Windows application log and SQLServer error log

1. Details on accessing the Windows application log are here: Viewing the Windows Application Log 

2. Details on accessing the SQL Server Error logs are here: View the SQL Server error log 

                                                   ii. See Monitoring and Tuning Tools for details on the different monitoring available.