Topic
  • 23 replies
  • Latest Post - ‏2012-09-10T11:55:25Z by bluey01
bluey01
bluey01
41 Posts

Pinned topic Web Console Scheduling stopped running jobs

‏2012-05-23T05:19:12Z |
Environ is Linux Redhat 5.7 64 bit DB2 9.7 and all software installed on the Linux server. I use a web browser on my Win7 PC for access.

I setup 3 jobs to run every night to copy some backup images and then do a new backup of 3 databases and this worked well for about a week. Today I checked and noticed that for 2 days now the jobs have not run. There is nothing in the Job history - so the jobs just stopped running. The schedules that the jobs are using are setup to run until the 31st May - so they should still be running.

Today I manually trigered one of the jobs and it ran successfully.

So 2 questions:
1. Has anyone seen this behviour and found a solution?
2. How does the schduler work and what can I check to see why its not running the jobs?
Updated on 2012-09-10T11:55:25Z at 2012-09-10T11:55:25Z by bluey01
  • SystemAdmin
    SystemAdmin
    1632 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-23T17:14:07Z  
    Hi,
    We had fixed an issuein the Web Console 3.1.1 reelease where there was trouble with jobs running weekly using Every other and every Friday/Sat/Sun options. So this coul dbe related. If you can attach the DS_System.log and JobManagerLogger.log files from the "logs" directory, we can take a look. Thanks.
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-24T04:54:53Z  
    Hi,
    We had fixed an issuein the Web Console 3.1.1 reelease where there was trouble with jobs running weekly using Every other and every Friday/Sat/Sun options. So this coul dbe related. If you can attach the DS_System.log and JobManagerLogger.log files from the "logs" directory, we can take a look. Thanks.
    OK - I will try and track down the logs. What does concern me is that if this is the preferred way to schedule backups (with Control Centre being deprecated), then it doesn't inspire confidence if it fails without any warnings after 5 or so days. This was a proof of concept to use for both test and prod environments.

    Should I be looking at some other approach at this stage if this new tool is not tested and proven?
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-24T04:59:56Z  
    • bluey01
    • ‏2012-05-24T04:54:53Z
    OK - I will try and track down the logs. What does concern me is that if this is the preferred way to schedule backups (with Control Centre being deprecated), then it doesn't inspire confidence if it fails without any warnings after 5 or so days. This was a proof of concept to use for both test and prod environments.

    Should I be looking at some other approach at this stage if this new tool is not tested and proven?
    DS System log below - not much too see....

    DS_System (139) May 3, 2012 4:03:16 PM INFO DBResourceManagerImpl: Initialization start...
    DS_System (139) May 3, 2012 4:03:16 PM INFO DBResourceManagerImpl: Checking for core schema presence...
    DS_System (139) May 3, 2012 4:03:17 PM INFO DBResourceManagerImpl: Core schema does not exist. Creating...
    DS_System (139) May 3, 2012 4:03:18 PM INFO
    DBResourceManagerImpl: Create core schema successfully
    DS_System (139) May 3, 2012 4:03:18 PM INFO DBResourceManagerImpl: Initialization complete...
    DS_System (168) May 3, 2012 4:03:18 PM INFO Test
    DS_System (172) May 3, 2012 4:03:23 PM INFO CDIHS0502I: The Health Summary repository started successfully.
    DS_System (172) May 3, 2012 4:03:23 PM INFO features: {EnableMetrics=false}
    DS_System (172) May 3, 2012 4:03:23 PM INFO CDIHS0500I: The Health Summary started successfully.

    #############################################################################################

    JobManagerLogger.0

    JobManagerLogger (618) May 4, 2012 8:17:47 AM ERROR Exception in run() of ShellScriptRunner
    java.lang.NullPointerException
    at com.jcraft.jsch.ChannelExec.setCommand(ChannelExec.java:63)
    at com.ibm.datatools.shared.services.util.ssh.SSHUtility.execCommand(SSHUtility.java:205)
    at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:152)
    at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
    at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
    at java.lang.Thread.run(Thread.java:736)
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-24T05:13:12Z  
    • bluey01
    • ‏2012-05-24T04:59:56Z
    DS System log below - not much too see....

    DS_System (139) May 3, 2012 4:03:16 PM INFO DBResourceManagerImpl: Initialization start...
    DS_System (139) May 3, 2012 4:03:16 PM INFO DBResourceManagerImpl: Checking for core schema presence...
    DS_System (139) May 3, 2012 4:03:17 PM INFO DBResourceManagerImpl: Core schema does not exist. Creating...
    DS_System (139) May 3, 2012 4:03:18 PM INFO
    DBResourceManagerImpl: Create core schema successfully
    DS_System (139) May 3, 2012 4:03:18 PM INFO DBResourceManagerImpl: Initialization complete...
    DS_System (168) May 3, 2012 4:03:18 PM INFO Test
    DS_System (172) May 3, 2012 4:03:23 PM INFO CDIHS0502I: The Health Summary repository started successfully.
    DS_System (172) May 3, 2012 4:03:23 PM INFO features: {EnableMetrics=false}
    DS_System (172) May 3, 2012 4:03:23 PM INFO CDIHS0500I: The Health Summary started successfully.

    #############################################################################################

    JobManagerLogger.0

    JobManagerLogger (618) May 4, 2012 8:17:47 AM ERROR Exception in run() of ShellScriptRunner
    java.lang.NullPointerException
    at com.jcraft.jsch.ChannelExec.setCommand(ChannelExec.java:63)
    at com.ibm.datatools.shared.services.util.ssh.SSHUtility.execCommand(SSHUtility.java:205)
    at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:152)
    at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
    at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
    at java.lang.Thread.run(Thread.java:736)
    The only other issue I can see is in the HealthSnapShotLogger which shows the error below. There are 8 databases where its OK, but not sure what database this is failing on - but I assume this shouldn't affect the Job schedule.

    ealthSnapshotLogger (35224) May 19, 2012 9:57:01 PM ERROR EXECUTE
    com.ibm.db2.jcc.am.SqlSyntaxErrorException: "DB2INST2" does not have the required authorization or privilege to perform operation "EXECUTE" on object "SYSPROC.MON_GET_CONNECTION".. SQLCODE=-551, SQLSTATE=42501, DRIVER=3.63.81
    at com.ibm.db2.jcc.am.fd.a(fd.java:679)
    at com.ibm.db2.jcc.am.fd.a(fd.java:60)
    at com.ibm.db2.jcc.am.fd.a(fd.java:127)
    at com.ibm.db2.jcc.am.yn.c(yn.java:2644)
    at com.ibm.db2.jcc.am.yn.d(yn.java:2632)
    at com.ibm.db2.jcc.am.yn.a(yn.java:2097)
    at com.ibm.db2.jcc.am.zn.a(zn.java:7201)
    at com.ibm.db2.jcc.t4.cb.h(cb.java:141)
    at com.ibm.db2.jcc.t4.cb.b(cb.java:41)
    at com.ibm.db2.jcc.t4.q.a(q.java:32)
    at com.ibm.db2.jcc.t4.sb.i(sb.java:135)
    at com.ibm.db2.jcc.am.yn.gb(yn.java:2066)
    at com.ibm.db2.jcc.am.zn.pc(zn.java:3450)
    at com.ibm.db2.jcc.am.zn.vc(zn.java:3582)
    at com.ibm.db2.jcc.am.zn.ic(zn.java:3048)
    at com.ibm.db2.jcc.am.zn.getParameterMetaData(zn.java:3013)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.getParameterMetaData(DelegatingPreparedStatement.java:212)
    at com.ibm.datatools.health.extn.DefaultSQLSnapshotService.updateSnapShotSQLParameters(DefaultSQLSnapshotService.java:118)
    at com.ibm.datatools.health.extn.DefaultSQLSnapshotService.prepareAndExecuteSelectStmt(DefaultSQLSnapshotService.java:299)
    at com.ibm.datatools.health.extn.DefaultSQLSnapshotService.executeSnapShotSQLAndInsert(DefaultSQLSnapshotService.java:242)
    at com.ibm.datatools.health.extn.DefaultSQLSnapshotService.collectSnapshot(DefaultSQLSnapshotService.java:526)
    at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:64)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:78)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:36)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at $Proxy130.collectSnapshot(Unknown Source)
    at com.ibm.datatools.health.extn.CaptureSnapshotAlertJob.run(CaptureSnapshotAlertJob.java:98)
    at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
    at java.lang.Thread.run(Thread.java:736)
  • SystemAdmin
    SystemAdmin
    1632 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-24T16:43:29Z  
    • bluey01
    • ‏2012-05-24T05:13:12Z
    The only other issue I can see is in the HealthSnapShotLogger which shows the error below. There are 8 databases where its OK, but not sure what database this is failing on - but I assume this shouldn't affect the Job schedule.

    ealthSnapshotLogger (35224) May 19, 2012 9:57:01 PM ERROR EXECUTE
    com.ibm.db2.jcc.am.SqlSyntaxErrorException: "DB2INST2" does not have the required authorization or privilege to perform operation "EXECUTE" on object "SYSPROC.MON_GET_CONNECTION".. SQLCODE=-551, SQLSTATE=42501, DRIVER=3.63.81
    at com.ibm.db2.jcc.am.fd.a(fd.java:679)
    at com.ibm.db2.jcc.am.fd.a(fd.java:60)
    at com.ibm.db2.jcc.am.fd.a(fd.java:127)
    at com.ibm.db2.jcc.am.yn.c(yn.java:2644)
    at com.ibm.db2.jcc.am.yn.d(yn.java:2632)
    at com.ibm.db2.jcc.am.yn.a(yn.java:2097)
    at com.ibm.db2.jcc.am.zn.a(zn.java:7201)
    at com.ibm.db2.jcc.t4.cb.h(cb.java:141)
    at com.ibm.db2.jcc.t4.cb.b(cb.java:41)
    at com.ibm.db2.jcc.t4.q.a(q.java:32)
    at com.ibm.db2.jcc.t4.sb.i(sb.java:135)
    at com.ibm.db2.jcc.am.yn.gb(yn.java:2066)
    at com.ibm.db2.jcc.am.zn.pc(zn.java:3450)
    at com.ibm.db2.jcc.am.zn.vc(zn.java:3582)
    at com.ibm.db2.jcc.am.zn.ic(zn.java:3048)
    at com.ibm.db2.jcc.am.zn.getParameterMetaData(zn.java:3013)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.getParameterMetaData(DelegatingPreparedStatement.java:212)
    at com.ibm.datatools.health.extn.DefaultSQLSnapshotService.updateSnapShotSQLParameters(DefaultSQLSnapshotService.java:118)
    at com.ibm.datatools.health.extn.DefaultSQLSnapshotService.prepareAndExecuteSelectStmt(DefaultSQLSnapshotService.java:299)
    at com.ibm.datatools.health.extn.DefaultSQLSnapshotService.executeSnapShotSQLAndInsert(DefaultSQLSnapshotService.java:242)
    at com.ibm.datatools.health.extn.DefaultSQLSnapshotService.collectSnapshot(DefaultSQLSnapshotService.java:526)
    at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:64)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:78)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:36)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at $Proxy130.collectSnapshot(Unknown Source)
    at com.ibm.datatools.health.extn.CaptureSnapshotAlertJob.run(CaptureSnapshotAlertJob.java:98)
    at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
    at java.lang.Thread.run(Thread.java:736)
    Hi,
    Pls. continue to use to the Web Console tool for Job Management and we will do our best to help you resolve any issues.
    The error in the HealthSnapshotLogger is not related to Job Manager.
    For the Executable/Shell Script type, we use SSH to connect to the remote database and execute the script. From the error in Job Manager Logger, it seems there was an exception in opening the channel.
    Can you pls. try to connect to the remote system(where the database resides) using ssh from the command line and try executing simple shell commands like:ps-ef;ls -l; and see if that works.
    Also if you create any new Shell script job with simple commands as above - does it run now or has it completely stopped?
    Thanks
    Dimple
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-25T02:34:24Z  
    Hi,
    Pls. continue to use to the Web Console tool for Job Management and we will do our best to help you resolve any issues.
    The error in the HealthSnapshotLogger is not related to Job Manager.
    For the Executable/Shell Script type, we use SSH to connect to the remote database and execute the script. From the error in Job Manager Logger, it seems there was an exception in opening the channel.
    Can you pls. try to connect to the remote system(where the database resides) using ssh from the command line and try executing simple shell commands like:ps-ef;ls -l; and see if that works.
    Also if you create any new Shell script job with simple commands as above - does it run now or has it completely stopped?
    Thanks
    Dimple
    The SSH error was when I first setup the environment. The scheduled jobs started running around the 14th May and stopped about 5 days later. I have also noticed that the job history files are set to be kept for 5 days - maybe this is more than a co-incidence.

    Today I triggered the jobs manually and they worked. I will wait for more updates. But in themean time I will delete the existing schedules and setup new ones and try it again. If it fails again, then I really cannot propose to use this tool in a Prod setting - it really needs to be robust enought to run correctly.
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-29T12:08:59Z  
    • bluey01
    • ‏2012-05-25T02:34:24Z
    The SSH error was when I first setup the environment. The scheduled jobs started running around the 14th May and stopped about 5 days later. I have also noticed that the job history files are set to be kept for 5 days - maybe this is more than a co-incidence.

    Today I triggered the jobs manually and they worked. I will wait for more updates. But in themean time I will delete the existing schedules and setup new ones and try it again. If it fails again, then I really cannot propose to use this tool in a Prod setting - it really needs to be robust enought to run correctly.
    Up to day 4 and so far so good. The next 2 days will show if it was a one off or not.
  • SystemAdmin
    SystemAdmin
    1632 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-29T22:23:38Z  
    • bluey01
    • ‏2012-05-29T12:08:59Z
    Up to day 4 and so far so good. The next 2 days will show if it was a one off or not.
    Hi,
    I think there is a bug related to deleting the schedules when the hsitory gets deleted. I see you mentioned that the History retention settings was also 5 days and then after 5 days the jobs did not run. Could pls. change the Log settings to Trace. Under the Open menu, go to Product Setup->Services. Select the "System logging service" and then click on Configure. A dialog will pop up and then select "Trace" from the drop down. The workaround would be to retain the success/failed record Indefinitely. Meanwhile, if you could pls. open a PMR, we can formally debug this , after the 5th day, you can attach the DS_System.log and we can also provide a patch for this issue.
    Thanks,
    Dimple
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-31T03:46:27Z  
    Hi,
    I think there is a bug related to deleting the schedules when the hsitory gets deleted. I see you mentioned that the History retention settings was also 5 days and then after 5 days the jobs did not run. Could pls. change the Log settings to Trace. Under the Open menu, go to Product Setup->Services. Select the "System logging service" and then click on Configure. A dialog will pop up and then select "Trace" from the drop down. The workaround would be to retain the success/failed record Indefinitely. Meanwhile, if you could pls. open a PMR, we can formally debug this , after the 5th day, you can attach the DS_System.log and we can also provide a patch for this issue.
    Thanks,
    Dimple
    And last night the jobs didn't run, so there is definitely a problem with scheduling tool - and I suspect dimplebhatia's suspicions are correct that its related to the history.

    Currently this is in a test environment, but I need something for Prod - so it looks like some other tool, the current version is not robust enough to go near Prod.

    Below is the stack trace form the Job Manager - I will also post some of the DS_Sytem.0 as well


    JobManagerLogger (190) May 30, 2012 11:03:11 AM ERROR Throwable exception in getConnection() or SQLUtil.delete() of deleteSchedAndHistory()
    org.eclipse.datatools.connectivity.ConnectionProfileException: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct4203011211http://3.63.81 A communication error occurred during operations on the connection's underlying socket, socket input stream,
    or socket output stream. Error location: Reply.fill() - insufficient data (-1). Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.makeNewConnection(DBConnectionManagementService.java:316)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:597)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:633)
    at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:64)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:78)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:36)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at $Proxy10.getConnection(Unknown Source)
    at com.ibm.datatools.common.repository.mgr.DatabaseProfileInstance.getConnection(DatabaseProfileInstance.java:75)
    at com.ibm.datatools.common.repository.mgr.RepositoryContributor.getConnection(RepositoryContributor.java:362)
    at com.ibm.datatools.common.repository.mgr.RepositoryContributor.getConnection(RepositoryContributor.java:369)
    at com.ibm.datatools.common.repository.mgr.SimpleSQLTableCRUD.getConnection(SimpleSQLTableCRUD.java:86)
    at com.ibm.datatools.jobmanager.JobManagerService.deleteSchedAndHistory(JobManagerService.java:1875)
    at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:64)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:78)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:36)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at $Proxy106.deleteSchedAndHistory(Unknown Source)
    at com.ibm.datatools.jobmanager.JobCleaner.doClean(JobCleaner.java:79)
    at com.ibm.datatools.jobmanager.JobCleaner.run(JobCleaner.java:42)
    Caused by: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct4203011211http://3.63.81 A communication error occurred during operations on the connection's underlying socket, socket input stream,
    or socket output stream. Error location: Reply.fill() - insufficient data (-1). Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
    at com.ibm.db2.jcc.am.fd.a(fd.java:321)
    at com.ibm.db2.jcc.t4.a.a(a.java:488)
    at com.ibm.db2.jcc.t4.a.a(a.java:483)
    at com.ibm.db2.jcc.t4.z.b(z.java:263)
    at com.ibm.db2.jcc.t4.z.c(z.java:289)
    at com.ibm.db2.jcc.t4.z.c(z.java:402)
    at com.ibm.db2.jcc.t4.z.v(z.java:1170)
    at com.ibm.db2.jcc.t4.ab.a(ab.java:47)
    at com.ibm.db2.jcc.t4.b.n(b.java:1321)
    at com.ibm.db2.jcc.t4.b.b(b.java:1193)
    at com.ibm.db2.jcc.t4.b.b(b.java:781)
    at com.ibm.db2.jcc.t4.b.b(b.java:753)
    at com.ibm.db2.jcc.t4.b.a(b.java:418)
    at com.ibm.db2.jcc.t4.b.a(b.java:393)
    at com.ibm.db2.jcc.t4.b.<init>(b.java:331)
    at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:231)
    at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:197)
    at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:472)
    at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:113)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnection.createConnectionInternal(DBConnection.java:298)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnection.createConnection(DBConnection.java:221)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.makeNewConnection(DBConnectionManagementService.java:280)
    ... 45 more
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-31T03:59:17Z  
    • bluey01
    • ‏2012-05-31T03:46:27Z
    And last night the jobs didn't run, so there is definitely a problem with scheduling tool - and I suspect dimplebhatia's suspicions are correct that its related to the history.

    Currently this is in a test environment, but I need something for Prod - so it looks like some other tool, the current version is not robust enough to go near Prod.

    Below is the stack trace form the Job Manager - I will also post some of the DS_Sytem.0 as well


    JobManagerLogger (190) May 30, 2012 11:03:11 AM ERROR Throwable exception in getConnection() or SQLUtil.delete() of deleteSchedAndHistory()
    org.eclipse.datatools.connectivity.ConnectionProfileException: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct4203011211http://3.63.81 A communication error occurred during operations on the connection's underlying socket, socket input stream,
    or socket output stream. Error location: Reply.fill() - insufficient data (-1). Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.makeNewConnection(DBConnectionManagementService.java:316)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:597)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:633)
    at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:64)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:78)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:36)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at $Proxy10.getConnection(Unknown Source)
    at com.ibm.datatools.common.repository.mgr.DatabaseProfileInstance.getConnection(DatabaseProfileInstance.java:75)
    at com.ibm.datatools.common.repository.mgr.RepositoryContributor.getConnection(RepositoryContributor.java:362)
    at com.ibm.datatools.common.repository.mgr.RepositoryContributor.getConnection(RepositoryContributor.java:369)
    at com.ibm.datatools.common.repository.mgr.SimpleSQLTableCRUD.getConnection(SimpleSQLTableCRUD.java:86)
    at com.ibm.datatools.jobmanager.JobManagerService.deleteSchedAndHistory(JobManagerService.java:1875)
    at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:64)
    at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:78)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:36)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at $Proxy106.deleteSchedAndHistory(Unknown Source)
    at com.ibm.datatools.jobmanager.JobCleaner.doClean(JobCleaner.java:79)
    at com.ibm.datatools.jobmanager.JobCleaner.run(JobCleaner.java:42)
    Caused by: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct4203011211http://3.63.81 A communication error occurred during operations on the connection's underlying socket, socket input stream,
    or socket output stream. Error location: Reply.fill() - insufficient data (-1). Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
    at com.ibm.db2.jcc.am.fd.a(fd.java:321)
    at com.ibm.db2.jcc.t4.a.a(a.java:488)
    at com.ibm.db2.jcc.t4.a.a(a.java:483)
    at com.ibm.db2.jcc.t4.z.b(z.java:263)
    at com.ibm.db2.jcc.t4.z.c(z.java:289)
    at com.ibm.db2.jcc.t4.z.c(z.java:402)
    at com.ibm.db2.jcc.t4.z.v(z.java:1170)
    at com.ibm.db2.jcc.t4.ab.a(ab.java:47)
    at com.ibm.db2.jcc.t4.b.n(b.java:1321)
    at com.ibm.db2.jcc.t4.b.b(b.java:1193)
    at com.ibm.db2.jcc.t4.b.b(b.java:781)
    at com.ibm.db2.jcc.t4.b.b(b.java:753)
    at com.ibm.db2.jcc.t4.b.a(b.java:418)
    at com.ibm.db2.jcc.t4.b.a(b.java:393)
    at com.ibm.db2.jcc.t4.b.<init>(b.java:331)
    at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:231)
    at com.ibm.db2.jcc.DB2SimpleDataSource.getConnection(DB2SimpleDataSource.java:197)
    at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:472)
    at com.ibm.db2.jcc.DB2Driver.connect(DB2Driver.java:113)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnection.createConnectionInternal(DBConnection.java:298)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnection.createConnection(DBConnection.java:221)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.makeNewConnection(DBConnectionManagementService.java:280)
    ... 45 more
    And here is the other log file. The job was scheduled to start at 2am. The log looks like it logged somthing, but there is no entry in the Job Hitory and it certainly didn't do the backup.


    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (207) May 31, 2012 2:00:00 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: 1337039129676.1337039129676-1 Trigger: jobmanager/1337039129676.1337039129676-1
    DS_System (207) May 31, 2012 2:00:00 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (207) May 31, 2012 2:00:00 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (207) May 31, 2012 2:00:00 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61197) May 31, 2012 2:01:11 AM TRACE SHS deleteAlerts(contributor, dsName)
    DS_System (61197) May 31, 2012 2:01:11 AM TRACE SHS deleteAlerts()
    DS_System (61197) May 31, 2012 2:01:11 AM TRACE SHS removeAlertsFromRepository(cats, dsName)
    DS_System (61197) May 31, 2012 2:01:11 AM DEBUG SHS - removing alerts from repository by datasource 'MYDB01-mytestserver1'
    DS_System (61196) May 31, 2012 2:01:11 AM TRACE ProcessRunner:run Task returned from execution task/SessionTASK-30343/SQW30343
    DS_System (61196) May 31, 2012 2:01:11 AM TRACE ProcessRunner: Instance execution of PI1292edc2de401e68_200 finished. Status: 2
    DS_System (61196) May 31, 2012 2:01:11 AM TRACE ProcessRunner:run finished run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@29982998 (procId: health.SnapshotMgtService.MYDB01-mytestserver1, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292edc2de401e68_200
    DS_System (200) May 31, 2012 2:01:11 AM TRACE ProcessManagerStatusImpl:execute Exiting - Blocking
    DS_System (138) May 31, 2012 2:03:46 AM TRACE DBConnectionProfileManager: syncConnections thread (re)starts
    DS_System (208) May 31, 2012 2:06:59 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: health.SnapshotMgtService.MYDBCNV1 Trigger: health/health.SnapshotMgtService.MYDBCNV1
    DS_System (208) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (208) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (208) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61198) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run started run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@50c350c3 (procId: health.SnapshotMgtService.MYDBCNV1, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292ee44ef09e300_208
    DS_System (61198) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created process instance com.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessInstanceImpl@164f164f (instId: PI1292ee44ef09e300_208, procId: health.SnapshotMgtService.MYDBCNV1, compId: health, startTime: 0, endTime: 0, status: 0)
    DS_System (61198) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created session into Task manager SQW30344
    DS_System (61198) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Added task into Task manager, About to execute task/SessionTASK-30344/SQW30344
    DS_System (203) May 31, 2012 2:06:59 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: health.SnapshotMgtService.MYDBDEV1 Trigger: health/health.SnapshotMgtService.MYDBDEV1
    DS_System (203) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (203) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (203) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61200) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run started run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@dc20dc2 (procId: health.SnapshotMgtService.MYDBDEV1, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292ee44fa99a250_203
    DS_System (61200) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created process instance com.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessInstanceImpl@5f855f85 (instId: PI1292ee44fa99a250_203, procId: health.SnapshotMgtService.MYDBDEV1, compId: health, startTime: 0, endTime: 0, status: 0)
    DS_System (61200) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created session into Task manager SQW30345
    DS_System (61200) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Added task into Task manager, About to execute task/SessionTASK-30345/SQW30345
    DS_System (204) May 31, 2012 2:06:59 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: health.SnapshotMgtService.MYDBINT1 Trigger: health/health.SnapshotMgtService.MYDBINT1
    DS_System (204) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (204) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (204) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61202) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run started run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@6d086d08 (procId: health.SnapshotMgtService.MYDBINT1, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292ee4502b47e60_204
    DS_System (61202) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created process instance com.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessInstanceImpl@33f833f8 (instId: PI1292ee4502b47e60_204, procId: health.SnapshotMgtService.MYDBINT1, compId: health, startTime: 0, endTime: 0, status: 0)
    DS_System (61202) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created session into Task manager SQW30346
    DS_System (61202) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Added task into Task manager, About to execute task/SessionTASK-30346/SQW30346
    DS_System (202) May 31, 2012 2:06:59 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: health.SnapshotMgtService.MYDBINT2 Trigger: health/health.SnapshotMgtService.MYDBINT2
    DS_System (202) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (202) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (202) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61204) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run started run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@5b4f5b4f (procId: health.SnapshotMgtService.MYDBINT2, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292ee450fe03e58_202
    DS_System (61204) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created process instance com.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessInstanceImpl@1c6f1c6f (instId: PI1292ee450fe03e58_202, procId: health.SnapshotMgtService.MYDBINT2, compId: health, startTime: 0, endTime: 0, status: 0)
    DS_System (61204) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created session into Task manager SQW30347
    DS_System (61204) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Added task into Task manager, About to execute task/SessionTASK-30347/SQW30347
    DS_System (61205) May 31, 2012 2:07:00 AM TRACE SHS - beginBatchUpdate()
    DS_System (61205) May 31, 2012 2:07:00 AM TRACE SHS - endBatchUpdate()
    DS_System (61205) May 31, 2012 2:07:00 AM TRACE SHS - beginBatchUpdate()
    DS_System (61205) May 31, 2012 2:07:00 AM TRACE SHS - endBatchUpdate()
    DS_System (61203) May 31, 2012 2:07:00 AM TRACE SHS - beginBatchUpdate()
  • SystemAdmin
    SystemAdmin
    1632 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-05-31T15:37:36Z  
    • bluey01
    • ‏2012-05-31T03:59:17Z
    And here is the other log file. The job was scheduled to start at 2am. The log looks like it logged somthing, but there is no entry in the Job Hitory and it certainly didn't do the backup.


    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - beginBatchUpdate()
    DS_System (61197) May 31, 2012 1:59:42 AM TRACE SHS - endBatchUpdate()
    DS_System (207) May 31, 2012 2:00:00 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: 1337039129676.1337039129676-1 Trigger: jobmanager/1337039129676.1337039129676-1
    DS_System (207) May 31, 2012 2:00:00 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (207) May 31, 2012 2:00:00 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (207) May 31, 2012 2:00:00 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61197) May 31, 2012 2:01:11 AM TRACE SHS deleteAlerts(contributor, dsName)
    DS_System (61197) May 31, 2012 2:01:11 AM TRACE SHS deleteAlerts()
    DS_System (61197) May 31, 2012 2:01:11 AM TRACE SHS removeAlertsFromRepository(cats, dsName)
    DS_System (61197) May 31, 2012 2:01:11 AM DEBUG SHS - removing alerts from repository by datasource 'MYDB01-mytestserver1'
    DS_System (61196) May 31, 2012 2:01:11 AM TRACE ProcessRunner:run Task returned from execution task/SessionTASK-30343/SQW30343
    DS_System (61196) May 31, 2012 2:01:11 AM TRACE ProcessRunner: Instance execution of PI1292edc2de401e68_200 finished. Status: 2
    DS_System (61196) May 31, 2012 2:01:11 AM TRACE ProcessRunner:run finished run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@29982998 (procId: health.SnapshotMgtService.MYDB01-mytestserver1, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292edc2de401e68_200
    DS_System (200) May 31, 2012 2:01:11 AM TRACE ProcessManagerStatusImpl:execute Exiting - Blocking
    DS_System (138) May 31, 2012 2:03:46 AM TRACE DBConnectionProfileManager: syncConnections thread (re)starts
    DS_System (208) May 31, 2012 2:06:59 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: health.SnapshotMgtService.MYDBCNV1 Trigger: health/health.SnapshotMgtService.MYDBCNV1
    DS_System (208) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (208) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (208) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61198) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run started run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@50c350c3 (procId: health.SnapshotMgtService.MYDBCNV1, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292ee44ef09e300_208
    DS_System (61198) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created process instance com.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessInstanceImpl@164f164f (instId: PI1292ee44ef09e300_208, procId: health.SnapshotMgtService.MYDBCNV1, compId: health, startTime: 0, endTime: 0, status: 0)
    DS_System (61198) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created session into Task manager SQW30344
    DS_System (61198) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Added task into Task manager, About to execute task/SessionTASK-30344/SQW30344
    DS_System (203) May 31, 2012 2:06:59 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: health.SnapshotMgtService.MYDBDEV1 Trigger: health/health.SnapshotMgtService.MYDBDEV1
    DS_System (203) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (203) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (203) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61200) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run started run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@dc20dc2 (procId: health.SnapshotMgtService.MYDBDEV1, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292ee44fa99a250_203
    DS_System (61200) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created process instance com.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessInstanceImpl@5f855f85 (instId: PI1292ee44fa99a250_203, procId: health.SnapshotMgtService.MYDBDEV1, compId: health, startTime: 0, endTime: 0, status: 0)
    DS_System (61200) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created session into Task manager SQW30345
    DS_System (61200) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Added task into Task manager, About to execute task/SessionTASK-30345/SQW30345
    DS_System (204) May 31, 2012 2:06:59 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: health.SnapshotMgtService.MYDBINT1 Trigger: health/health.SnapshotMgtService.MYDBINT1
    DS_System (204) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (204) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (204) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61202) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run started run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@6d086d08 (procId: health.SnapshotMgtService.MYDBINT1, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292ee4502b47e60_204
    DS_System (61202) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created process instance com.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessInstanceImpl@33f833f8 (instId: PI1292ee4502b47e60_204, procId: health.SnapshotMgtService.MYDBINT1, compId: health, startTime: 0, endTime: 0, status: 0)
    DS_System (61202) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created session into Task manager SQW30346
    DS_System (61202) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Added task into Task manager, About to execute task/SessionTASK-30346/SQW30346
    DS_System (202) May 31, 2012 2:06:59 AM TRACE >>>> ProcessRunner.run() about to call processManager.execute() with procId: health.SnapshotMgtService.MYDBINT2 Trigger: health/health.SnapshotMgtService.MYDBINT2
    DS_System (202) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:execute Entering
    DS_System (202) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Entering
    DS_System (202) May 31, 2012 2:06:59 AM TRACE ProcessManagerStatusImpl:getProcessDefinition Eexiting
    DS_System (61204) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run started run for procId/instIdcom.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessDefinitionImpl@5b4f5b4f (procId: health.SnapshotMgtService.MYDBINT2, compId: health, taskClass: com.ibm.datatools.health.extn.CaptureSnapshotAlertJob, taskBundle: com.ibm.datatools.health.extn, userInstId: null, createUser: null, scheduled: false)/PI1292ee450fe03e58_202
    DS_System (61204) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created process instance com.ibm.datatools.common.processmgmt.processMgmt.impl.ProcessInstanceImpl@1c6f1c6f (instId: PI1292ee450fe03e58_202, procId: health.SnapshotMgtService.MYDBINT2, compId: health, startTime: 0, endTime: 0, status: 0)
    DS_System (61204) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Created session into Task manager SQW30347
    DS_System (61204) May 31, 2012 2:06:59 AM TRACE ProcessRunner:run Added task into Task manager, About to execute task/SessionTASK-30347/SQW30347
    DS_System (61205) May 31, 2012 2:07:00 AM TRACE SHS - beginBatchUpdate()
    DS_System (61205) May 31, 2012 2:07:00 AM TRACE SHS - endBatchUpdate()
    DS_System (61205) May 31, 2012 2:07:00 AM TRACE SHS - beginBatchUpdate()
    DS_System (61205) May 31, 2012 2:07:00 AM TRACE SHS - endBatchUpdate()
    DS_System (61203) May 31, 2012 2:07:00 AM TRACE SHS - beginBatchUpdate()
    Hi,
    We know the fix for your issue. The internal process definitions created for the schedules also gets deleted after the history records get purged. As I mentioned, the workaround is to select the "Indefinitely" option in the Job History settings dialog. If you can open a PMR, we can work with you and our support team to provide a patch for this issue.
    Pls. let us know. Thanks.

    Dimple
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-06-01T00:03:37Z  
    Hi,
    We know the fix for your issue. The internal process definitions created for the schedules also gets deleted after the history records get purged. As I mentioned, the workaround is to select the "Indefinitely" option in the Job History settings dialog. If you can open a PMR, we can work with you and our support team to provide a patch for this issue.
    Pls. let us know. Thanks.

    Dimple
    I have done the testing for you and you know the problem, so why do I have to spend time opening a PMR with IBM - I have to look after our business needs - I don't have time to raise PMRs.

    This DataStudio platform needs some serious work by IBM to make it usable. Today I set it up on a Windows server and from the web console nothing runs because it cannot get a SSH connection to windows. And I am not installing more stuff on

    The potential of this tool is terrific and when I first looked at it I was excited. However it is so flakey and hard to get working consistently that I cannot consider this tool until:
    a) the tool installs and works seamlessly with all *nix and Win environments
    b) the tool's functionality works as described

    Maybe I will revist the tool in the next release
  • Sriram_Europa
    Sriram_Europa
    68 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-06-01T03:43:10Z  
    • bluey01
    • ‏2012-06-01T00:03:37Z
    I have done the testing for you and you know the problem, so why do I have to spend time opening a PMR with IBM - I have to look after our business needs - I don't have time to raise PMRs.

    This DataStudio platform needs some serious work by IBM to make it usable. Today I set it up on a Windows server and from the web console nothing runs because it cannot get a SSH connection to windows. And I am not installing more stuff on

    The potential of this tool is terrific and when I first looked at it I was excited. However it is so flakey and hard to get working consistently that I cannot consider this tool until:
    a) the tool installs and works seamlessly with all *nix and Win environments
    b) the tool's functionality works as described

    Maybe I will revist the tool in the next release
    Hi

    I am very sorry that you are having so much trouble here.

    The suggestion to work with IBM support via the PMR mechanism was to see if we can resolve your problems much faster. For example, IBM support could provide you with the location for a fix, if the "retain history indefinitely" workaround was not convenient. This would be in the form of a patch for your current installation, rather than having to wait for a fix in a future release.

    I am hoping that you will continue using this product and provide us with feedback in enhancing it and better suiting your scenarios.

    Please feel free to contact me via email (my address is my dev works ID) & we will do our best to resolve this as soon as possible.

    Regards
    Sriram
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-06-04T00:54:47Z  
    Hi

    I am very sorry that you are having so much trouble here.

    The suggestion to work with IBM support via the PMR mechanism was to see if we can resolve your problems much faster. For example, IBM support could provide you with the location for a fix, if the "retain history indefinitely" workaround was not convenient. This would be in the form of a patch for your current installation, rather than having to wait for a fix in a future release.

    I am hoping that you will continue using this product and provide us with feedback in enhancing it and better suiting your scenarios.

    Please feel free to contact me via email (my address is my dev works ID) & we will do our best to resolve this as soon as possible.

    Regards
    Sriram
    Hi Sriram
    I will test the "retain history indefinitely" workaround in the test environment and see if it provides some stability. Hopefully we will see a new relase soon with this issue rectified.

    I notice that my previous post was not complete.

    I have also just installed DataStudio Web Console on a Windows server, however I cannot get a connection (from my PC) as it requires a SSH connection. Is this correct - that the only way to connect to the database server is via SSH?

    What about if I run the Web console on the database server itself - does it still require a SSH connection or not?

    I am about to test the same type of scheduling on a windows server.
  • Sriram_Europa
    Sriram_Europa
    68 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-06-05T00:29:36Z  
    • bluey01
    • ‏2012-06-04T00:54:47Z
    Hi Sriram
    I will test the "retain history indefinitely" workaround in the test environment and see if it provides some stability. Hopefully we will see a new relase soon with this issue rectified.

    I notice that my previous post was not complete.

    I have also just installed DataStudio Web Console on a Windows server, however I cannot get a connection (from my PC) as it requires a SSH connection. Is this correct - that the only way to connect to the database server is via SSH?

    What about if I run the Web console on the database server itself - does it still require a SSH connection or not?

    I am about to test the same type of scheduling on a windows server.
    Thanks for trying the workaround. Please contact me by email should you want to pursue the patch option too.

    As regards to your other question - yes - even if DSWC is installed on the same system as the Database server - it will require the use of SSH. Its generally best to install DSWC on a system separate from the Database server so as not to impact the performance of that system. Its also common to run the DSWC server as one (lower privileged) user and execute ssh as a completely different user - we rely on sshd to provide for the login environment needed.
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-06-13T01:59:45Z  
    Thanks for trying the workaround. Please contact me by email should you want to pursue the patch option too.

    As regards to your other question - yes - even if DSWC is installed on the same system as the Database server - it will require the use of SSH. Its generally best to install DSWC on a system separate from the Database server so as not to impact the performance of that system. Its also common to run the DSWC server as one (lower privileged) user and execute ssh as a completely different user - we rely on sshd to provide for the login environment needed.
    Again the jobs have stopped running and I am trying to track down the reason.

    The server (linux Redhat 5.7 64 bit)had an unplanned reboot. So the first thing was that the Web Console was not running, so restarted this. Since then none of the jobs are running.

    The HealthSnapshotLogger is showing success for connecting tot he various databases.

    The dsserver.log is throwing errors from the day of the outage, but nothing since. Looks like the scheduler is no longer scheduling (see log output below).

    I suspect that if I create new schedules it will start working. Again it is frustating to use this tool. There is not a lot of info on how this should be setup to run consistently and how to recover from a server restart. How should this tool be configured and what needs to be redone if the DB2 instance is stopped for some maintenance or a server outage is required. What steps are requried to stop and start DSWC gracefully (and is this documented anywhere?).

    Also what is missing from the tool is something on the Web console to show the state of the tool itself - something to show any errors or what jobs are actively scheduled.

    =======================================================
    1 http://QuartzScheduler_com.ibm.datatools.services.scheduler-NON_CLUSTERED_MisfireHandler com.ibm.datatools.services.scheduler.impl.ExtJobStoreTX - MisfireHandler: Error handling misfires: Failed to obtain DB connection from data source 'scheduler': java.sql.SQLException: MetaDBConnectionProvider: Error getting connection, Reason:com.ibm.db2.jcc.am.SqlException
    org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'scheduler': java.sql.SQLException: MetaDBConnectionProvider: Error getting connection, Reason:com.ibm.db2.jcc.am.SqlException See nested exception: java.sql.SQLException: MetaDBConnectionProvider: Error getting connection, Reason:com.ibm.db2.jcc.am.SqlException
    at com.ibm.datatools.services.scheduler.impl.ExtJobStoreTX.getConnectionCustom(ExtJobStoreTX.java:39)
    at com.ibm.datatools.services.scheduler.impl.ExtJobStoreTX.getNonManagedTXConnection(ExtJobStoreTX.java:27)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3107)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
    Caused by: java.sql.SQLException: MetaDBConnectionProvider: Error getting connection, Reason:com.ibm.db2.jcc.am.SqlException
    at com.ibm.datatools.services.scheduler.impl.MetaDBConnectionProvider.getConnection(MetaDBConnectionProvider.java:48)
    at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:109)
    at com.ibm.datatools.services.scheduler.impl.ExtJobStoreTX.getConnectionCustom(ExtJobStoreTX.java:35)
    ... 4 more
    2012-06-11 14:44:39 com.ibm.datatools.services.log.impl.DSLoggerImpl::logError Thread-4049
    SEVERE [ ERROR com.ibm.datatools.common.repository.mgr.RepositoryException: org.eclipse.datatools.connectivity.ConnectionProfileException: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct42043115503.63.81] Exception java.net.ConnectException: Error opening socket to server localhost/127.0.0.1 on port 50,000 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001
    2012-06-11 14:44:39 com.ibm.datatools.services.log.impl.DSLoggerImpl::logError Thread-4049
    SEVERE http:// ERROR Throwable exception in getConnection() or SQLUtil.delete() of deleteSchedAndHistory()
    org.eclipse.datatools.connectivity.ConnectionProfileException: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct4204311550http://3.63.81 Exception java.net.ConnectException: Error opening socket to server localhost/127.0.0.1 on port 50,000 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.makeNewConnection(DBConnectionManagementService.java:316)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:597)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:633)
    at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
    org.eclipse.datatools.connectivity.ConnectionProfileException: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct4204311550http://3.63.81 Exception java.net.ConnectException: Error opening socket to server localhost/127.0.0.1 on port 50,000 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.makeNewConnection(DBConnectionManagementService.java:316)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:597)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:633)
  • Sriram_Europa
    Sriram_Europa
    68 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-06-13T17:21:20Z  
    • bluey01
    • ‏2012-06-13T01:59:45Z
    Again the jobs have stopped running and I am trying to track down the reason.

    The server (linux Redhat 5.7 64 bit)had an unplanned reboot. So the first thing was that the Web Console was not running, so restarted this. Since then none of the jobs are running.

    The HealthSnapshotLogger is showing success for connecting tot he various databases.

    The dsserver.log is throwing errors from the day of the outage, but nothing since. Looks like the scheduler is no longer scheduling (see log output below).

    I suspect that if I create new schedules it will start working. Again it is frustating to use this tool. There is not a lot of info on how this should be setup to run consistently and how to recover from a server restart. How should this tool be configured and what needs to be redone if the DB2 instance is stopped for some maintenance or a server outage is required. What steps are requried to stop and start DSWC gracefully (and is this documented anywhere?).

    Also what is missing from the tool is something on the Web console to show the state of the tool itself - something to show any errors or what jobs are actively scheduled.

    =======================================================
    1 http://QuartzScheduler_com.ibm.datatools.services.scheduler-NON_CLUSTERED_MisfireHandler com.ibm.datatools.services.scheduler.impl.ExtJobStoreTX - MisfireHandler: Error handling misfires: Failed to obtain DB connection from data source 'scheduler': java.sql.SQLException: MetaDBConnectionProvider: Error getting connection, Reason:com.ibm.db2.jcc.am.SqlException
    org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'scheduler': java.sql.SQLException: MetaDBConnectionProvider: Error getting connection, Reason:com.ibm.db2.jcc.am.SqlException See nested exception: java.sql.SQLException: MetaDBConnectionProvider: Error getting connection, Reason:com.ibm.db2.jcc.am.SqlException
    at com.ibm.datatools.services.scheduler.impl.ExtJobStoreTX.getConnectionCustom(ExtJobStoreTX.java:39)
    at com.ibm.datatools.services.scheduler.impl.ExtJobStoreTX.getNonManagedTXConnection(ExtJobStoreTX.java:27)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3107)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
    Caused by: java.sql.SQLException: MetaDBConnectionProvider: Error getting connection, Reason:com.ibm.db2.jcc.am.SqlException
    at com.ibm.datatools.services.scheduler.impl.MetaDBConnectionProvider.getConnection(MetaDBConnectionProvider.java:48)
    at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:109)
    at com.ibm.datatools.services.scheduler.impl.ExtJobStoreTX.getConnectionCustom(ExtJobStoreTX.java:35)
    ... 4 more
    2012-06-11 14:44:39 com.ibm.datatools.services.log.impl.DSLoggerImpl::logError Thread-4049
    SEVERE [ ERROR com.ibm.datatools.common.repository.mgr.RepositoryException: org.eclipse.datatools.connectivity.ConnectionProfileException: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct42043115503.63.81] Exception java.net.ConnectException: Error opening socket to server localhost/127.0.0.1 on port 50,000 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001
    2012-06-11 14:44:39 com.ibm.datatools.services.log.impl.DSLoggerImpl::logError Thread-4049
    SEVERE http:// ERROR Throwable exception in getConnection() or SQLUtil.delete() of deleteSchedAndHistory()
    org.eclipse.datatools.connectivity.ConnectionProfileException: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct4204311550http://3.63.81 Exception java.net.ConnectException: Error opening socket to server localhost/127.0.0.1 on port 50,000 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.makeNewConnection(DBConnectionManagementService.java:316)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:597)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:633)
    at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
    org.eclipse.datatools.connectivity.ConnectionProfileException: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: jcct4204311550http://3.63.81 Exception java.net.ConnectException: Error opening socket to server localhost/127.0.0.1 on port 50,000 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.makeNewConnection(DBConnectionManagementService.java:316)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:597)
    at com.ibm.datatools.common.db.connmgt.impl.DBConnectionManagementService.getConnection(DBConnectionManagementService.java:633)
    The messages below:
    Exception java.net.ConnectException: Error opening socket to server localhost/127.0.0.1 on port 50,000 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001

    indicates that when the job was about to be executed - there was no connection to the repository database. This happened at: 2012-06-11 14:44:39.

    I suspect that the repository database either did not come back up in time after the reboot or was down at this particular time. Since all the job metadata is stored in the repository database - jobs cannot execute without the repository database.

    Apart from the logs, in the Product Setup -> Configuration Repository page, you should see signs that indicate if the repository database is active. By choosing "Select Repository Database" and clicking on "Test connection", you can confirm if the repository DB is still reachable.
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-06-14T00:48:31Z  
    The messages below:
    Exception java.net.ConnectException: Error opening socket to server localhost/127.0.0.1 on port 50,000 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001

    indicates that when the job was about to be executed - there was no connection to the repository database. This happened at: 2012-06-11 14:44:39.

    I suspect that the repository database either did not come back up in time after the reboot or was down at this particular time. Since all the job metadata is stored in the repository database - jobs cannot execute without the repository database.

    Apart from the logs, in the Product Setup -> Configuration Repository page, you should see signs that indicate if the repository database is active. By choosing "Select Repository Database" and clicking on "Test connection", you can confirm if the repository DB is still reachable.
    I did check that yesterday and the repository was active (see cut and paste below). Soeven if the database was not active for a little time, its been active for a few days now, but still no jobs are running.

    Repository database detailsHost Name Port Number Database
    localhost 50000 ZZDSWC
    The repository database ZZDSWC is active.

    Any other suggestions for this?
  • Sriram_Europa
    Sriram_Europa
    68 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-06-15T03:07:50Z  
    • bluey01
    • ‏2012-06-14T00:48:31Z
    I did check that yesterday and the repository was active (see cut and paste below). Soeven if the database was not active for a little time, its been active for a few days now, but still no jobs are running.

    Repository database detailsHost Name Port Number Database
    localhost 50000 ZZDSWC
    The repository database ZZDSWC is active.

    Any other suggestions for this?
    I am not sure why it was not able to connect to the repository database at that time. Around the same time frame -did you happen to see any alerts (Open -> Health -> Alerts) that could potentially give us a clue here ? An internal IBM user had issues when their DB credential had expired and they needed to reset it to resume job executions - but the logs clearly indicated that state with Alerts as well as in Health Alert configuration panels. Other than that I haven't seen SQLState situations like yours.

    DSWC records diag information about what happens in every service in the logs/ files - so we can figure out why exactly the scheduler service had these connection/misfire errors.
    It might be best if you open up a support call and they can collect more info about your installation here.
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-06-20T00:29:50Z  
    I am not sure why it was not able to connect to the repository database at that time. Around the same time frame -did you happen to see any alerts (Open -> Health -> Alerts) that could potentially give us a clue here ? An internal IBM user had issues when their DB credential had expired and they needed to reset it to resume job executions - but the logs clearly indicated that state with Alerts as well as in Health Alert configuration panels. Other than that I haven't seen SQLState situations like yours.

    DSWC records diag information about what happens in every service in the logs/ files - so we can figure out why exactly the scheduler service had these connection/misfire errors.
    It might be best if you open up a support call and they can collect more info about your installation here.
    Thanks for the update. At this stage I think I will have to look at other methods of automating tasks - which is a shame, as the potential of the tool is terrific, but I hvae not been able to get scheduled tasks to run for more than a week without some sort of failure.

    After a month of trying to use this tool here is a summary of my impressions:

    The Good
    1. The tool has some great features - being able to run SQL, scripts etc
    2. The ability to schedule and chain jobs is good
    3. A central web interface for multiple servers is great

    The Bad
    4. The scheduler is not reliable and has failed for several different reasons
    5. The scheduler has some know bugs that need fixing
    6. The WebConsole interface is clumsy - especially for adding schedules and notifications - it needs some work on it
    7. Documentation on the tool is very light

    The Ugly
    8. SSH requried for windows servers - this is painful
    9. The WebConsole and scheduler do not recover from systems outages well

    I will wait in anticipation for the next release to see if this tool gets closer to realising its full potential. In the mean time it looks like CRON and Windows scheduler.
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-09-05T07:15:04Z  
    • bluey01
    • ‏2012-06-20T00:29:50Z
    Thanks for the update. At this stage I think I will have to look at other methods of automating tasks - which is a shame, as the potential of the tool is terrific, but I hvae not been able to get scheduled tasks to run for more than a week without some sort of failure.

    After a month of trying to use this tool here is a summary of my impressions:

    The Good
    1. The tool has some great features - being able to run SQL, scripts etc
    2. The ability to schedule and chain jobs is good
    3. A central web interface for multiple servers is great

    The Bad
    4. The scheduler is not reliable and has failed for several different reasons
    5. The scheduler has some know bugs that need fixing
    6. The WebConsole interface is clumsy - especially for adding schedules and notifications - it needs some work on it
    7. Documentation on the tool is very light

    The Ugly
    8. SSH requried for windows servers - this is painful
    9. The WebConsole and scheduler do not recover from systems outages well

    I will wait in anticipation for the next release to see if this tool gets closer to realising its full potential. In the mean time it looks like CRON and Windows scheduler.
    As an update to the Web Scheduler - having kept the history, the scheduled jobs have been running quite well for 2 months now. Manually having to remove the history is a pain though.

    I would love to use this for Windows as well - but the SSH requriement is a real drawback. As we work in a virtual environment - Windows servers are created very quickly. To then have to install SSH tools just to get to DB2 is not very elegant.

    Is there any update as to when a new release will be available?
  • Sriram_Europa
    Sriram_Europa
    68 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-09-07T18:32:22Z  
    • bluey01
    • ‏2012-09-05T07:15:04Z
    As an update to the Web Scheduler - having kept the history, the scheduled jobs have been running quite well for 2 months now. Manually having to remove the history is a pain though.

    I would love to use this for Windows as well - but the SSH requriement is a real drawback. As we work in a virtual environment - Windows servers are created very quickly. To then have to install SSH tools just to get to DB2 is not very elegant.

    Is there any update as to when a new release will be available?
    Hi

    We hope to have an updated release end of this year - possibly in December.

    In the meantime do have a patch for the history removal issue that you can unzip on top of a DSWC v3.1.1. If you are not able to get this via a PMR to IBM support for some reason - I can send it out separately to you (my dev works ID is also my email address).

    As regards to the SSH requirement on windows - I agree that it is an irritant - and to make matters worse, Cygwin's SSHd behaves oddly in some cases too. Unfortunately - the secure communications provided by SSH is something we have come to rely on & any replacements would have to be as good (security wise). Executing shell scripts remotely is risky by itself - but doing it securely - without letting viruses & trojans hijack your system remotely is absolutely critical.

    While we do require SSH for the DB2 CLP or Shell script executions - we wouldn't need it for just plain SQL. Just SQL is limiting - but there are surprisingly quite a bit of things you can do with ADMIN_CMD & other stored procedures. But I guess this doesn't quite work in your case ?

    DB2 v10.1 now includes a specialized SSH server as part of this installation. For DSWC - we are looking at to support this server and make all dependency on cygwin & cygwin's sshd optional. So when you provision your virtual environment's DB2 instances - would it be possible for you to have the IBM SSH server installed as well ?

    http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/topic/com.ibm.db2.luw.qb.server.doc/doc/t0059009.html
  • bluey01
    bluey01
    41 Posts

    Re: Web Console Scheduling stopped running jobs

    ‏2012-09-10T11:55:25Z  
    Hi

    We hope to have an updated release end of this year - possibly in December.

    In the meantime do have a patch for the history removal issue that you can unzip on top of a DSWC v3.1.1. If you are not able to get this via a PMR to IBM support for some reason - I can send it out separately to you (my dev works ID is also my email address).

    As regards to the SSH requirement on windows - I agree that it is an irritant - and to make matters worse, Cygwin's SSHd behaves oddly in some cases too. Unfortunately - the secure communications provided by SSH is something we have come to rely on & any replacements would have to be as good (security wise). Executing shell scripts remotely is risky by itself - but doing it securely - without letting viruses & trojans hijack your system remotely is absolutely critical.

    While we do require SSH for the DB2 CLP or Shell script executions - we wouldn't need it for just plain SQL. Just SQL is limiting - but there are surprisingly quite a bit of things you can do with ADMIN_CMD & other stored procedures. But I guess this doesn't quite work in your case ?

    DB2 v10.1 now includes a specialized SSH server as part of this installation. For DSWC - we are looking at to support this server and make all dependency on cygwin & cygwin's sshd optional. So when you provision your virtual environment's DB2 instances - would it be possible for you to have the IBM SSH server installed as well ?

    http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/topic/com.ibm.db2.luw.qb.server.doc/doc/t0059009.html
    THanks for the update.

    Will have a look for the patch or email you as suggested.

    Will also look at the SSH server as well.