Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
11 replies Latest Post - ‏2012-11-02T14:28:20Z by SystemAdmin
SystemAdmin
SystemAdmin
1632 Posts
ACCEPTED ANSWER

Pinned topic DSWC returns "Auth cancel" on scheduled jobs

‏2012-10-24T16:20:17Z |
Hello,

We're running DSWC 3.1.1 on zLinux.
All our scheduled jobs ended with "Auth cancel" last night. Here is an extract of JobManagaerLogger.0:
JobManagerLogger (478) Oct 23, 2012 10:30:00 PM TRACE Calling ShellScriptRunner.executeCommand()
JobManagerLogger (478) Oct 23, 2012 10:30:00 PM ERROR JSchException in run() of ShellScriptRunner
com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:451)
at com.ibm.datatools.shared.services.util.ssh.SSHUtility.connectToHost(SSHUtility.java:101)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:149)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
at java.lang.Thread.run(Thread.java:736)

JobManagerLogger (478) Oct 23, 2012 10:30:00 PM ERROR Exception in run() of ShellScriptRunner
com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:451)
at com.ibm.datatools.shared.services.util.ssh.SSHUtility.connectToHost(SSHUtility.java:101)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:149)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
at java.lang.Thread.run(Thread.java:736)

JobManagerLogger (542) Oct 23, 2012 10:45:00 PM TRACE Calling ShellScriptRunner.executeCommand()
JobManagerLogger (542) Oct 23, 2012 10:45:03 PM ERROR JSchException in run() of ShellScriptRunner
com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:451)
at com.ibm.datatools.shared.services.util.ssh.SSHUtility.connectToHost(SSHUtility.java:101)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:149)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
at java.lang.Thread.run(Thread.java:736)

JobManagerLogger (542) Oct 23, 2012 10:45:03 PM ERROR Exception in run() of ShellScriptRunner
com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:451)
at com.ibm.datatools.shared.services.util.ssh.SSHUtility.connectToHost(SSHUtility.java:101)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:149)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
at java.lang.Thread.run(Thread.java:736)

JobManagerLogger (576) Oct 23, 2012 11:00:00 PM TRACE Calling ShellScriptRunner.executeCommand()
JobManagerLogger (576) Oct 23, 2012 11:00:05 PM ERROR JSchException in run() of ShellScriptRunner
com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:451)
at com.ibm.datatools.shared.services.util.ssh.SSHUtility.connectToHost(SSHUtility.java:101)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:149)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
at java.lang.Thread.run(Thread.java:736)

JobManagerLogger (576) Oct 23, 2012 11:00:05 PM ERROR Exception in run() of ShellScriptRunner
com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:451)
at com.ibm.datatools.shared.services.util.ssh.SSHUtility.connectToHost(SSHUtility.java:101)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:149)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
at java.lang.Thread.run(Thread.java:736)

JobManagerLogger (640) Oct 23, 2012 11:15:00 PM TRACE Calling ShellScriptRunner.executeCommand()
JobManagerLogger (640) Oct 23, 2012 11:15:00 PM ERROR JSchException in run() of ShellScriptRunner
com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:451)
at com.ibm.datatools.shared.services.util.ssh.SSHUtility.connectToHost(SSHUtility.java:101)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:149)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
at java.lang.Thread.run(Thread.java:736)

JobManagerLogger (640) Oct 23, 2012 11:15:00 PM ERROR Exception in run() of ShellScriptRunner
com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:451)
at com.ibm.datatools.shared.services.util.ssh.SSHUtility.connectToHost(SSHUtility.java:101)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.executeCommand(ShellScriptRunner.java:149)
at com.ibm.datatools.jobmanager.execution.ShellScriptRunner.run(ShellScriptRunner.java:89)
at com.ibm.datatools.sqwruntime.taskmgr.core.impl.TaskRunner.run(TaskRunner.java:54)
at java.lang.Thread.run(Thread.java:736)

But when we manually ran the jobs this morning, they all ended OK.

Any idea?

Thanks
M
Updated on 2012-11-02T14:28:20Z at 2012-11-02T14:28:20Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    1632 Posts
    ACCEPTED ANSWER

    Re: DSWC returns "Auth cancel" on scheduled jobs

    ‏2012-10-26T20:26:32Z  in response to SystemAdmin
    Please see the attached for JobManagerLogger.0
    • SystemAdmin
      SystemAdmin
      1632 Posts
      ACCEPTED ANSWER

      Re: DSWC returns "Auth cancel" on scheduled jobs

      ‏2012-10-28T01:22:35Z  in response to SystemAdmin
      Thanks for attaching the logs. At the end of the log file, I see the ShellScript was successful sinc ethe exit staus was 0. Earlier it had the Auth fail exception. So is the Shell Script working now?

      ================================
      JobManagerLogger (4188) Oct 26, 2012 5:24:24 PM TRACE Calling ShellScriptRunner.executeCommand()
      JobManagerLogger (4188) Oct 26, 2012 5:24:26 PM TRACE Calling sshSession.execCommand() in ShellScriptRunner.executeCommand()/db2bu-remote/db2scripts/db2_database_backup.sh cms db2cms > /db2bu/cms/backups/db2_database_backup.out
      JobManagerLogger (4188) Oct 26, 2012 5:24:54 PM TRACE Finished sshSession.execCommand() in ShellScriptRunner.executeCommand():exitStatus: 0CmdOutput:
      JobManagerLogger (4195) Oct 26, 2012 5:25:49 PM TRACE Calling ShellScriptRunner.executeCommand()
      JobManagerLogger (4195) Oct 26, 2012 5:25:50 PM TRACE Calling sshSession.execCommand() in ShellScriptRunner.executeCommand()/db2bu-remote/db2scripts/db2_database_backup.sh cmdb db2cmdb > /db2bu/cmdb/backups/db2_database_backup.out
      JobManagerLogger (4202) Oct 26, 2012 5:25:57 PM TRACE Calling ShellScriptRunner.executeCommand()
      JobManagerLogger (4202) Oct 26, 2012 5:26:00 PM TRACE Calling sshSession.execCommand() in ShellScriptRunner.executeCommand()/db2bu-remote/db2scripts/db2_database_backup.sh tbsm db2tbsm > /db2bu/tbsm/backups/db2_database_backup.out
      JobManagerLogger (4209) Oct 26, 2012 5:26:05 PM TRACE Calling ShellScriptRunner.executeCommand()
      JobManagerLogger (4209) Oct 26, 2012 5:26:11 PM TRACE Calling sshSession.execCommand() in ShellScriptRunner.executeCommand()/db2bu-remote/db2scripts/db2_database_backup.sh tipdb db2ncim > /db2bu/tipdb/backups/db2_database_backup.out
      JobManagerLogger (4216) Oct 26, 2012 5:26:13 PM TRACE Calling ShellScriptRunner.executeCommand()
      JobManagerLogger (4216) Oct 26, 2012 5:26:19 PM TRACE Calling sshSession.execCommand() in ShellScriptRunner.executeCommand()/db2bu-remote/db2scripts/db2_database_backup.sh ncim db2ncim
      JobManagerLogger (4223) Oct 26, 2012 5:26:21 PM TRACE Calling ShellScriptRunner.executeCommand()
      JobManagerLogger (4223) Oct 26, 2012 5:26:31 PM TRACE Calling sshSession.execCommand() in ShellScriptRunner.executeCommand()/db2bu-remote/db2scripts/db2_database_backup.sh monitor db2conn > /db2bu/monitor/backups/db2_database_backup.out
      JobManagerLogger (4209) Oct 26, 2012 5:27:06 PM TRACE Finished sshSession.execCommand() in ShellScriptRunner.executeCommand():exitStatus: 0CmdOutput:
      JobManagerLogger (4223) Oct 26, 2012 5:28:47 PM TRACE Finished sshSession.execCommand() in ShellScriptRunner.executeCommand():exitStatus: 0CmdOutput:
      JobManagerLogger (4195) Oct 26, 2012 5:31:58 PM TRACE Finished sshSession.execCommand() in ShellScriptRunner.executeCommand():exitStatus: 0CmdOutput:
      JobManagerLogger (4202) Oct 26, 2012 5:32:02 PM TRACE Finished sshSession.execCommand() in ShellScriptRunner.executeCommand():exitStatus: 0CmdOutput:

      =============
      Also see some warnings that some schedules are not active : so pls. ensure the schedules are active.
      ==========
      JobManagerLogger (3812) Oct 19, 2012 6:30:00 PM WARNING the job schedule was not kicked off since it was not enabled/activated.
      JobManagerLogger (3870) Oct 19, 2012 6:45:00 PM WARNING the job schedule was not kicked off since it was not enabled/activated.
      • SystemAdmin
        SystemAdmin
        1632 Posts
        ACCEPTED ANSWER

        Re: DSWC returns "Auth cancel" on scheduled jobs

        ‏2012-10-29T13:29:05Z  in response to SystemAdmin
        Hello Dimple,

        The "successful" messages were when we reran the failed jobs manually.
        The schedules are active - we temporarily suspended them only Oct 19.

        Im summary, scheduled shell scripts still failed aith "auth cancelled".

        M
        • SystemAdmin
          SystemAdmin
          1632 Posts
          ACCEPTED ANSWER

          Re: DSWC returns "Auth cancel" on scheduled jobs

          ‏2012-10-29T13:59:08Z  in response to SystemAdmin
          Hello Dimple,

          Additional info if it helps: only the ones scheduled to run on zLinux failed every night. The one on Linux worked fine.

          M
          • SystemAdmin
            SystemAdmin
            1632 Posts
            ACCEPTED ANSWER

            Re: DSWC returns "Auth cancel" on scheduled jobs

            ‏2012-10-29T16:49:14Z  in response to SystemAdmin
            Hi,
            FYI..For z/OS databases, we support SQL Only Script Job type. We do not support Shell Script. Can we verify that the ssh daemon is up and running at the scheduled time. Maybe you could try to schedule it while you are around to make sure the SSH setup is working fine that time and also you can manually run the same joba gainst the same database at that time.

            Thanks,
            Dimple
            • SystemAdmin
              SystemAdmin
              1632 Posts
              ACCEPTED ANSWER

              Re: DSWC returns "Auth cancel" on scheduled jobs

              ‏2012-10-29T18:21:44Z  in response to SystemAdmin
              Hello Dimple,

              This is zLinux not zOS.
              We will try changing the schedule time to sometimes between 9and 5 as you recommended and see what happens.

              M
              • SystemAdmin
                SystemAdmin
                1632 Posts
                ACCEPTED ANSWER

                Re: DSWC returns "Auth cancel" on scheduled jobs

                ‏2012-10-29T18:53:49Z  in response to SystemAdmin
                Hi Dimple,

                We change the schedule to run the jobs just now - same result "Auth cancel". They work when they're manually run therefore I don't think shell cript doesn't work on zLinuz as you mentioned.
                Any other idea?

                M
                • Sriram_Europa
                  Sriram_Europa
                  68 Posts
                  ACCEPTED ANSWER

                  Re: DSWC returns "Auth cancel" on scheduled jobs

                  ‏2012-10-29T23:09:42Z  in response to SystemAdmin
                  Hi - when you mean "manually" - do you mean by clicking on the "Run Job" (now) button or ssh from the command line ? If that succeeds but a scheduled job fails - it most likely something to do with the credentials used. ("Auth Cancel" is an exception thrown by the SSH layer)
                  When you schedule the job - did you select a DB and provide a specific user credential just for that particular schedule ? (for an example see - attachment singledbconn.jpg)

                  Or did you simply pick one or more DBs from the list without providing a different credential for that scheduled run ? In that case - the 'default' credential specified in the Databases page would be used. Select the DB & then click Edit & verify details. Click on Test Connection in the dialog itself to confirm that the credentials are fine.

                  For other diagnostics - do you see any alerts produced in the Health Summary or Alerts->Alerts List panels - with regards to this database ? "unreachable" or "cannot connect.." etc. messages ?
                  • SystemAdmin
                    SystemAdmin
                    1632 Posts
                    ACCEPTED ANSWER

                    Re: DSWC returns "Auth cancel" on scheduled jobs

                    ‏2012-10-30T15:27:21Z  in response to Sriram_Europa
                    Hi I work with Mai and we have both been trying to get Data Studio working for some time. To answer your questions these jobs are using a specific userid and password to the database, the connections have been tested and work. Not only that but this failure is not just for one database/server but for many. Note re-running the job from job history works, also they work when we use the "run job" without the scheduler. I could be wrong but this indicates to me that the setup and authorities are correct as is the script it is executing on the server. The problem seems to be the scheduler and how it is trying to authenticate to ssh on the server. Note that scheduling works on other Linux systems just not on the 6 zLinux ones we are try to set up maintenance for.
                    On the database page all connections test fine, they also use the same id we specified in the job.
                    No diagnostic info is seen in the Health Summary or Alerts.

                    Thanks Rob
                    • Sriram_Europa
                      Sriram_Europa
                      68 Posts
                      ACCEPTED ANSWER

                      Re: DSWC returns "Auth cancel" on scheduled jobs

                      ‏2012-11-02T07:55:17Z  in response to SystemAdmin
                      Hi

                      The "run job" (now) works exactly the same way as the scheduled executions - both invoke ssh in the same manner. So - this is a bit odd. It it fails for one with an Auth Cancel exception - it should fail for the other as well. If there were firewall issues - then that would affect both runs. Can you confirm that the "run job" actual;y ran successfully ? Perhaps it too failed & there is some other reason why it appears to have succeeded ..

                      One key difference here is that with the scheduler can use a different credential to run the job instead of the credential used in the "Databases" page. Try editing that schedule entry that failed and select the "multiple database" option - pick the database that you want the script to run against (this will use the same credential as that recorded in the Databases page). I attached a snapshot of the select Multiple Databases panel
                      My next thought here is that the data associated with the actual "schedule" is incorrect for some reason. If you contact IBM support - they can work with you to extract the schedule data from your repository for analysis. (its not a good idea to share that kind of info in a public forum). Via IBM support - we can also share other tools and diagnostics collectors to identify exactly where the problem is.
                      If the schedule data itself is corrupted, you could try to create a new schedule for this job to execute, say a few minutes later and see if that works fine. or a create new simple job with a new schedule using the "multiple databases" selection.
                      • SystemAdmin
                        SystemAdmin
                        1632 Posts
                        ACCEPTED ANSWER

                        Re: DSWC returns "Auth cancel" on scheduled jobs

                        ‏2012-11-02T14:28:20Z  in response to Sriram_Europa
                        Hi,

                        Yes, I have confirmed the "run job" now actually did what it was supposed to do. I.E. the backup with some backup information was produced. I rebuild the failing Executable/Shell Script jobs as CLP and defaulted the user. Those jobs worked last night with no problems. I suspect that when we use a specific user when selecting a database for the schedule something happens. For now the CLP will do but we are missing some information we like to collect with the backup.

                        Question: Is there anyway of telling if a schedule uses a specific id after it has been created ?

                        I know under schedules I can edit the schedule and see the database but I have to select the database to see what type of user connection was used. That seems to show me specific id in some cases but I am not sure if it is showing the current value or just requesting me to redefine the connection. We mostly used specific user and only to a specific database in the past. Now it is looking like that was not the best way to do it. By defaulting to the default connection id our problems seem to be somewhat solved.
                        I will let you know Monday if there are any more issues.

                        Rob