Troubleshooting
Problem
Execution of schedule reports /jobs fails resulting in error "No Managed Connections" resulting in failure of all subsequent reports/ jobs
Symptom
Scheduled reports/ jobs fail to execute with below error :
ERROR:
07:28:40,522 ERROR [reportgroup.RGPHandler] Error checking if entity exists:org.jboss.util.NestedSQLException:
No ManagedConnections available within configured blocking timeout ( 30000 [ms] ); - nested throwable:
(javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] ))
07:28:40,522 ERROR [reportgroup.RGPHandler] Error checking if entity exists:org.jboss.util.NestedSQLException:
No ManagedConnections available within configured blocking timeout ( 30000 [ms] ); - nested throwable:
(javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] ))
Above error is encountered in respective component logs ( wmtm for pweb reports, summary log, sbh log ).
It impacts subsequent reports/ jobs and unavailability of aggregated data to TNPM end users.
Cause
No Managed Connections issue occurs when all connections are already in-use.
Any new reports/jobs which are getting triggered will try to get a DB connection and if connection pool do not have sufficient connections , then it results in No Managed Connections error.
Any new reports/jobs which are getting triggered will try to get a DB connection and if connection pool do not have sufficient connections , then it results in No Managed Connections error.
Below are the possible Causes for this error:
1. No. of jobs (reports, sbh, summary etc) which require connections are more than the available pooled connections in the system.
2. All the connections are occupied and underlying SQL's occupying these connections are slow due to which connections are not getting released to pool hence new reports which are trying to create connections are getting failed.
3. There might be one session which is blocking all the connections hence all the connections are occupied and any new reports or sql connections will throw error.
3. There might be one session which is blocking all the connections hence all the connections are occupied and any new reports or sql connections will throw error.
Environment
Product Version: TNPM Wireless Version 1.4.2, 1.4.3,1.4.4
Operating System: AIX; Linux; Solaris
Operating System: AIX; Linux; Solaris
Diagnosing The Problem
1. Check total No. of jobs (reports, sbh, summary etc) which require connections are sufficient than the available pooled connections in the system.
2. Checked whether all the connections are occupied and this connections are slow due to which connections are not getting released to pool hence new reports which are trying to create connections are getting failed.
3. There might be one session which is blocking all the connections hence all the connections are occupied and any new reports or sql connections will throw error. Consult your DBA to find any session which could be blocking other sessions. With details of the blocking session try to backtrack which report / job is resulting in the issue.
3. There might be one session which is blocking all the connections hence all the connections are occupied and any new reports or sql connections will throw error. Consult your DBA to find any session which could be blocking other sessions. With details of the blocking session try to backtrack which report / job is resulting in the issue.
Resolving The Problem
This issue can be Resolved by increasing pool connections size.
Below are the steps for increasing pool connections.
Changes need to be done in all 3 files.
1) /appl/virtuo/as/server/default/deploy/jms/jms-ds.xml
Increase by 20
<max-pool-size>40</max-pool-size>
2) /appl/virtuo/as/server/default/deploy_core/na-defds-ds.xml
Increase by 30
3) /appl/virtuo/as/server/default/deploy/hsqldb-ds.xml
Increase by 30
4) /appl/virtuo/as/server/default/deploy_core/na-alarm-ds.xml
Increase by 30
To make the changes effective, "as" service needs to be restarted.
Below are the steps for increasing pool connections.
Changes need to be done in all 3 files.
1) /appl/virtuo/as/server/default/deploy/jms/jms-ds.xml
Increase by 20
<max-pool-size>40</max-pool-size>
2) /appl/virtuo/as/server/default/deploy_core/na-defds-ds.xml
Increase by 30
3) /appl/virtuo/as/server/default/deploy/hsqldb-ds.xml
Increase by 30
4) /appl/virtuo/as/server/default/deploy_core/na-alarm-ds.xml
Increase by 30
To make the changes effective, "as" service needs to be restarted.
If issue is not resolved by increasing the pool connections, then check on below suggestion.
• Check the SID which is blocking other sessions.
• Use below queries for identifying the blocking sessions.
• alter session set nls_Date_format = 'DD-MON-YY HH24:MI:SS';
1. Identify blocking session:
select l1.sid, ' IS BLOCKING ', l2.sid
from v$lock l1, v$lock l2
where l1.block =1 and l2.request > 0
and l1.id1=l2.id1
and l1.id2=l2.id2;
You will get output like below :
364 IS BLOCKING 2711
2. Put the ids obtained from above query in below query:
select * from
v$session s , v$sql s1
where s1.sql_id =s.sql_id
and s.sid in (, ,);
• Once you identified the blocking SID investigate the SID from the database queries.
try to capture SQL ID’s stuck and tables which are getting locked at that time of issue.
• If No blocking session is captured then check the no of reports, sbh and summary running to identify the trend of reports.
• See if there are any long running reports which are causing slowness in database and consuming more time than its regular time.
• alter session set nls_Date_format = 'DD-MON-YY HH24:MI:SS';
1. Identify blocking session:
select l1.sid, ' IS BLOCKING ', l2.sid
from v$lock l1, v$lock l2
where l1.block =1 and l2.request > 0
and l1.id1=l2.id1
and l1.id2=l2.id2;
You will get output like below :
364 IS BLOCKING 2711
2. Put the ids obtained from above query in below query:
select * from
v$session s , v$sql s1
where s1.sql_id =s.sql_id
and s.sid in (, ,);
• Once you identified the blocking SID investigate the SID from the database queries.
try to capture SQL ID’s stuck and tables which are getting locked at that time of issue.
• If No blocking session is captured then check the no of reports, sbh and summary running to identify the trend of reports.
• See if there are any long running reports which are causing slowness in database and consuming more time than its regular time.
• You can use below queries to find out the no of reports, SBH, summary and other database contents.
1. alter session set nls_date_format= 'DD-MM-YY HH24:MI:SS'
2. select sbh_name, last_run, last_run_result, LAST_RUN_SECS, last_run_text from sbh_definition def, sbh_history hist where def.sbh_id = hist.sbh_id;
3. select distinct a.report_id, a.schedule_id, a.report_name, b.job_id, c.task_id, to_char(c.tstamp,'dd-MM-yyyy hh24:mi:ss'), to_char(c.started,'dd-MM-yyyy hh24:mi:ss'), to_char(c.ended,'dd-MM-yyyy hh24:mi:ss'), round((c.ended - c.started) * 86400) duration, c.status_code, c.message
from pe_report_schedule_vw a, active_job b, task_status c
where a.schedule_id = b.schedule_id
and b.job_id = c.job_id
and c.tstamp <= sysdate
and c.message != 'Scheduled to run'
and c.tstamp >= sysdate - 1
order by nvl(to_char(c.started,'dd-MM-yyyy hh24:mi:ss'),to_char(c.tstamp,'dd-MM-yyyy hh24:mi:ss')) desc, to_char(c.tstamp,'dd-MM-yyyy hh24:mi:ss') desc, c.task_id asc;
4. select H.SUMMARY_HISTORY_ID, H.SUMMARY_ID, D.SUMMARY_NAME, H.SUMMARY_INTERVAL, H.FIRST_AVAIL, H.LAST_AVAIL,H.LAST_RUN_START,H.LAST_RUN_END, (H.LAST_RUN_END - H.LAST_RUN_START)*24*60 AS difference_in_minutes, H.LAST_RUN_RESULT,H.LAST_RUN_TEXT,H.LAST_SUCCESSFUL_RUN
from summary_history H,summary_definition D
where H.summary_id=D.summary_id;
5. select *
from pe_report_schedule_vw a, active_job b, task_status c
where a.schedule_id = b.schedule_id
and b.job_id = c.job_id
and c.tstamp <= sysdate
and c.message != 'Scheduled to run'
and c.tstamp >= sysdate - 1
order by nvl(to_char(c.started,'dd-MM-yyyy hh24:mi:ss'),to_char(c.tstamp,'dd-MM-yyyy hh24:mi:ss')) desc, to_char(c.tstamp,'dd-MM-yyyy hh24:mi:ss') desc, c.task_id asc;
6. select *
from summary_history H,summary_definition D
where H.summary_id=D.summary_id;
Use below queries to find long running sessions ( need to be executed when issue is encountered )
7. select *
from v$session s, v$sql q
where s.sql_id = q.sql_id;
8. SELECT *
FROM v$session_longops sl
INNER JOIN v$sql sq ON sq.sql_id = sl.sql_id
INNER JOIN v$session s ON sl.SID = s.SID AND sl.serial# = s.serial#
WHERE time_remaining > 0;
9. select * from V$SQL_MONITOR;
10. select * from v$session_longops s, v$sql s2
where s.sql_id = s2.sql_id;
11. select * from
v$active_session_history s, v$sql s2 where s.sql_id = s2.sql_id;
12. select *
from v$session b, v$process a
where
b.paddr = a.addr
and type='USER' ;
From the report query identify the number of connections at the time of issue and compare to other time in the day.
Consult your DBA if a SQL session which is blocking other sessions and backtrack to the job which is executing the SQL. For time-being you may disable the job and investigate the cause.
Consult your DBA if a SQL session which is blocking other sessions and backtrack to the job which is executing the SQL. For time-being you may disable the job and investigate the cause.
As a workaround for blocking sessions , if the SQL is executed by any of the application job you can try restarting "as", which will reset the pool connection and also end the blocking session.
Document Location
Worldwide
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSKGHX","label":"Tivoli Netcool Performance Manager for Wireless"},"ARM Category":[],"Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Product Synonym
TNPM WIRELESS;TNPMW
Was this topic helpful?
Document Information
Modified date:
02 April 2020
UID
ibm15736861