IC5Notice: We have upgraded developerWorks Community to the latest version of IBM Connections. For more information, read our upgrade FAQ.
Topic
  • 10 replies
  • Latest Post - ‏2012-09-18T08:59:45Z by beach_bum
devguru99
devguru99
5 Posts

Pinned topic Datastage Controller problem

‏2012-09-18T06:19:43Z |
Hi all,

I have a sequencer which calls about 13 jobs in parallel.
Recently the sequencer has been failing with the below error.
Controller problem: Error calling DSRunJob(siiLoadTCAppointment), code=-14
Timed out while waiting for an event
The sequencer runs parallely with another sequencer with 35 jobs in parallel.
The sequence had been running without any such issues for years.
Configuration of the system is below:
Quad core(Xeon /2.800GHz); 4 GB RAM

Recently we have cleared old logs from each of the jobs.
This was because there used to be a delay in starting jobs.
This had improved the performance, but all of a sudden this issue started.
Is there any relation between the two?
Updated on 2012-09-18T08:59:45Z at 2012-09-18T08:59:45Z by beach_bum
  • beach_bum
    beach_bum
    49 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T06:39:14Z  
    So you have 48 jobs running in parallel on 4 cores? Are they server or px? If px, how many nodes are defined in the config file?

    In any case, you got what is an error message that suggests your server is out of resources, so something did change in the environment. These can be any number of things like OS patch or drivers, new programs comming in, amount of data processed, poor job logic, job changes - sometimes people change an enterprise stage to a connector and they claim they haven't changed anything, etc. You may want to investigate that.

    No matter the result of the above, I would take another look at sequencing these jobs so not all of them are run in parallel as you will likely end up with a faster overall process.

    Also, your last sentence suggests that your environment is not fully optimized (configured) so you might want to have a chat with your OS admin about that.
  • beach_bum
    beach_bum
    49 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T06:41:01Z  
    • beach_bum
    • ‏2012-09-18T06:39:14Z
    So you have 48 jobs running in parallel on 4 cores? Are they server or px? If px, how many nodes are defined in the config file?

    In any case, you got what is an error message that suggests your server is out of resources, so something did change in the environment. These can be any number of things like OS patch or drivers, new programs comming in, amount of data processed, poor job logic, job changes - sometimes people change an enterprise stage to a connector and they claim they haven't changed anything, etc. You may want to investigate that.

    No matter the result of the above, I would take another look at sequencing these jobs so not all of them are run in parallel as you will likely end up with a faster overall process.

    Also, your last sentence suggests that your environment is not fully optimized (configured) so you might want to have a chat with your OS admin about that.
    Btw, how did you clear the job logs from the jobs?
  • devguru99
    devguru99
    5 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T06:50:26Z  
    • beach_bum
    • ‏2012-09-18T06:39:14Z
    So you have 48 jobs running in parallel on 4 cores? Are they server or px? If px, how many nodes are defined in the config file?

    In any case, you got what is an error message that suggests your server is out of resources, so something did change in the environment. These can be any number of things like OS patch or drivers, new programs comming in, amount of data processed, poor job logic, job changes - sometimes people change an enterprise stage to a connector and they claim they haven't changed anything, etc. You may want to investigate that.

    No matter the result of the above, I would take another look at sequencing these jobs so not all of them are run in parallel as you will likely end up with a faster overall process.

    Also, your last sentence suggests that your environment is not fully optimized (configured) so you might want to have a chat with your OS admin about that.
    Muy query here is..

    The jobs used to run fine for more than 10 years now.
    The problem began when i cleared 9 year old logs for each of the jobs.

    Logs were cleared using the clear log option in Datastage director.
  • beach_bum
    beach_bum
    49 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T07:01:18Z  
    • devguru99
    • ‏2012-09-18T06:50:26Z
    Muy query here is..

    The jobs used to run fine for more than 10 years now.
    The problem began when i cleared 9 year old logs for each of the jobs.

    Logs were cleared using the clear log option in Datastage director.
    If logs were cleared from the director, then this should not have caused you the issue.

    If you've already investigated everything above (no one on the forums can do that for you) and you've followed the correct way of cleaning the job logs, I suggest you raise a PMR with IBM.
  • devguru99
    devguru99
    5 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T07:46:37Z  
    • beach_bum
    • ‏2012-09-18T07:01:18Z
    If logs were cleared from the director, then this should not have caused you the issue.

    If you've already investigated everything above (no one on the forums can do that for you) and you've followed the correct way of cleaning the job logs, I suggest you raise a PMR with IBM.
    Thanks a lot for your help.

    The bigger problem is, i am using datastage 6.0 and is out of support.
  • beach_bum
    beach_bum
    49 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T08:30:13Z  
    • devguru99
    • ‏2012-09-18T07:46:37Z
    Thanks a lot for your help.

    The bigger problem is, i am using datastage 6.0 and is out of support.
    Auch ... perhaps you should've led with that.

    Have you tried running only up to 2 or 3 jobs in parallel? If so, have you actually checked that absolutely nothing has changed on the environment (OS, database, amount of data, hardware)?
  • devguru99
    devguru99
    5 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T08:45:32Z  
    • beach_bum
    • ‏2012-09-18T08:30:13Z
    Auch ... perhaps you should've led with that.

    Have you tried running only up to 2 or 3 jobs in parallel? If so, have you actually checked that absolutely nothing has changed on the environment (OS, database, amount of data, hardware)?
    Thanks for your help with this.

    I am working on a support project for a client.
    So i cant do much about the version here.

    Also the same job definition runs on DEV server. There are no issues over there.

    Will you be able to suggest any ways to trace the root cause of this error?
  • beach_bum
    beach_bum
    49 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T08:54:03Z  
    • devguru99
    • ‏2012-09-18T08:45:32Z
    Thanks for your help with this.

    I am working on a support project for a client.
    So i cant do much about the version here.

    Also the same job definition runs on DEV server. There are no issues over there.

    Will you be able to suggest any ways to trace the root cause of this error?
    To be honest, even if it does work in Dev, it seems like the design can be improved upon.

    This message as far as I can tell means the server does not have enough resources for some reason.

    Is Dev and Prod identical in every aspect (including data volumes)?
  • devguru99
    devguru99
    5 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T08:57:06Z  
    • beach_bum
    • ‏2012-09-18T08:54:03Z
    To be honest, even if it does work in Dev, it seems like the design can be improved upon.

    This message as far as I can tell means the server does not have enough resources for some reason.

    Is Dev and Prod identical in every aspect (including data volumes)?
    The design has been there for 10 years now :):)...
    Change is risky...

    Any way to trace the issue will be helpful.
  • beach_bum
    beach_bum
    49 Posts

    Re: Datastage Controller problem

    ‏2012-09-18T08:59:45Z  
    • devguru99
    • ‏2012-09-18T08:57:06Z
    The design has been there for 10 years now :):)...
    Change is risky...

    Any way to trace the issue will be helpful.
    I think I told you a few ways a couple of times, but I missed the outcomes in your replies.

    Sadly, I know of no way to fix this without doing some grunt work. I'll be sure to let you know if I think of something, buddy :)

    Cheers,
    A