Queuing caused by high thread usage

If threads are the most constrained resource, the agent requirements (that is, the degree) of the concurrent queries is causing the queuing. To understand which activities are consuming threading resources, use the monitor query described here.

Use the MON_GET_ACTIVITY table function to determine the threading requirements for each currently executing statement. This table function also reports information about the current state of the query (EXECUTING OR IDLE) and identification information.
The thread requirements for each query are determined by the query degree, which is reported by the effective_query_degree monitor element.

WITH LOADTRGT(LOADTRGT) AS (SELECT MAX(VALUE) FROM SYSIBMADM.DBCFG WHERE NAME = 'wlm_agent_load_trgt'),
     CPUINFO(CPUS_PER_HOST) AS (SELECT MAX(CPU_ONLINE / CPU_HMT_DEGREE) FROM TABLE(ENV_GET_SYSTEM_RESOURCES())),
     PARTINFO(PART_PER_HOST) AS (SELECT COUNT(*) PART_PER_HOST FROM TABLE(DB_MEMBERS()) AS T WHERE T.MEMBER_TYPE = 'D' GROUP BY HOST_NAME FETCH FIRST 1 ROWS ONLY )
     SELECT A.APPLICATION_HANDLE,
            A.UOW_ID,
            A.ACTIVITY_ID,
            A.LOCAL_START_TIME,
            A.ACTIVITY_STATE,
            EFFECTIVE_QUERY_DEGREE AS THREAD_DEMAND,
            DEC(FLOAT(EFFECTIVE_QUERY_DEGREE) / (FLOAT(LOADTRGT) * FLOAT(CPUS_PER_HOST) / FLOAT(PART_PER_HOST))*100,5,2) PCT_THREAD_DEMAND,
            TIMESTAMPDIFF(2, (CURRENT_TIMESTAMP - A.LOCAL_START_TIME)) AS TOTAL_RUNTIME,
            B.APPLICATION_NAME,
            B.SESSION_AUTH_ID,
            A.MEMBER,
            SUBSTR(A.STMT_TEXT, 1, 512) AS STATEMENT_TEXT
FROM TABLE(MON_GET_ACTIVITY(NULL, -2)) AS A,
     TABLE(MON_GET_CONNECTION(NULL,-2)) AS B,
     LOADTRGT,
     CPUINFO,
     PARTINFO
WHERE A.APPLICATION_HANDLE = B.APPLICATION_HANDLE AND
      A.MEMBER = B.MEMBER AND
      A.ACTIVITY_STATE IN ('EXECUTING', 'IDLE') AND
      A.MEMBER = A.COORD_PARTITION_NUM AND
      A.ADM_BYPASSED = 0

The following sample output indicates the degree and percent thread utilization for each of the currently executing statements.

APPLICATION_HANDLE   UOW_ID      ACTIVITY_ID LOCAL_START_TIME           ACTIVITY_STATE   THREAD_DEMAND    PCT_THREAD_DEMAND TOTAL_RUNTIME APPLICATION_NAME  SESSION_AUTH_ID   MEMBER STATEMENT_TEXT 
-------------------- ----------- ----------- -------------------------- ---------------- ---------------- ----------------- ------------- ------------------ ---------------- ------ ---------------------------------------------------------------------------------------------------------
                 894           5           1 2019-05-07-11.28.50.275631 EXECUTING                       8              2.94           141 db2bp              USER1                 0 select count(*) from syscat.tables, syscat.indexes, syscat.workloads, syscat.tables, syscat.tables                                                                                                                                                                                                                                                                                                                                                                                                                              
                 799          53           1 2019-05-07-11.31.11.411797 EXECUTING                       1              0.36             0 db2bp              USER1                 0 WITH LOADTRGT(LOADTRGT) AS (SELECT MAX(VALUE) FROM SYSIBMADM.DBCFG WHERE NAME = 'wlm_agent_load_trgt'), ...

Use the output to determine if any queries are running with an unexpectedly high query degree. If there is enough CPU available on the host where the database resides (e.g. historically OS tooling shows the CPU is not saturated), consider moderately increasing the wlm_agent_load_trgt database configuration parameter to allow more threads to execute concurrently.