Topic
2 replies Latest Post - ‏2012-12-07T03:37:27Z by SystemAdmin
SystemAdmin
SystemAdmin
603 Posts
ACCEPTED ANSWER

Pinned topic Oozie workflow error (stream exceeds limit [2048]

‏2012-11-28T15:42:26Z |
Hello:

I am trying to run a simple oozie workflow executing a shell script. This shell script just calls a java program that is actually working properly. Once the java program execution ends I was expecting to see the workflow status changing from RUNNING to SUCCESFUL, but this is not happening.

This is the content of the workflow log:

2012-11-28 16:31:34,221 DEBUG StartCommand:70 - USER[?] GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] STARTED SignalCommand for jobid=0000006-121128124244941-oozie-biad-W, actionId=null
2012-11-28 16:31:34,223 DEBUG LiteWorkflowInstance:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] Starting job
2012-11-28 16:31:34,223 DEBUG LiteWorkflowInstance:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] Signaling job execution path [/] signal value ::start::
2012-11-28 16:31:34,224 DEBUG LiteWorkflowInstance:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] Exiting node ::start:: with transition[/#SSHJob]
2012-11-28 16:31:34,224 DEBUG LiteWorkflowInstance:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] Signaling job execution path [/] signal value ::synch::
2012-11-28 16:31:34,226 DEBUG LiteWorkflowStoreService:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] Creating action for node SSHJob
2012-11-28 16:31:34,232 DEBUG StartCommand:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] Updated the workflow status to 0000006-121128124244941-oozie-biad-W status =RUNNING
2012-11-28 16:31:34,232 DEBUG StartCommand:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] ENDED SignalCommand for jobid=0000006-121128124244941-oozie-biad-W, actionId=null
2012-11-28 16:31:34,252 DEBUG ActionStartCommand:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] STARTED ActionStartCommand for wf actionId=0000006-121128124244941-oozie-biad-W@SSHJob
2012-11-28 16:31:34,253 WARN ActionStartCommand:96 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION[-] ***0000006-121128124244941-oozie-biad-W@SSHJob***In call()....status=PREP
2012-11-28 16:31:34,257 DEBUG ActionStartCommand:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob Start, name SSHJob type ssh configuration
biadmin@bigdata
/tmp/testingestor/start-ingestor.sh
--config-file /tmp/testingestor/basic.conf

2012-11-28 16:31:34,257 INFO SshActionExecutor:84 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob start() begins
2012-11-28 16:31:34,258 INFO SshActionExecutor:84 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob Attempting to copy ssh base scripts to remote host biadmin@bigdata
2012-11-28 16:31:35,183 INFO SshActionExecutor:84 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob start() ends
2012-11-28 16:31:35,184 WARN ActionStartCommand:96 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob ***0000006-121128124244941-oozie-biad-W@SSHJob***Action status=RUNNING
2012-11-28 16:31:35,186 WARN ActionStartCommand:96 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob ***0000006-121128124244941-oozie-biad-W@SSHJob***Action updated in DB!
2012-11-28 16:31:35,186 DEBUG ActionStartCommand:70 - USERbiadmin GROUPbiadmin] TOKEN[ APPssh-wf JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob ENDED ActionStartCommand for wf actionId=0000006-121128124244941-oozie-biad-W@SSHJob, jobId=0000006-121128124244941-oozie-biad-W
2012-11-28 16:32:48,367 DEBUG CallbackServlet:70 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob Received a CallbackServlet.doPost() with query string id=0000006-121128124244941-oozie-biad-W@SSHJob&status=OK&
2012-11-28 16:32:48,367 INFO CallbackServlet:84 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob callback for action 0000006-121128124244941-oozie-biad-W@SSHJob
2012-11-28 16:32:48,369 ERROR CallbackServlet:108 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB0000006-121128124244941-oozie-biad-W ACTION0000006-121128124244941-oozie-biad-W@SSHJob URLPOST http://bigdata:8280/oozie/v0/callback?id=0000006-121128124244941-oozie-biad-W@SSHJob&status=OK& error, stream exceeds limit 2,048
java.lang.IllegalArgumentException: stream exceeds limit 2,048
at org.apache.oozie.util.IOUtils.getReaderAsString(IOUtils.java:79)
at org.apache.oozie.servlet.CallbackServlet.doPost(CallbackServlet.java:128)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:281)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.geronimo.tomcat.valve.DefaultSubjectValve.invoke(DefaultSubjectValve.java:56)
at org.apache.geronimo.tomcat.GeronimoStandardContext$SystemMethodValve.invoke(GeronimoStandardContext.java:406)
at org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke(GeronimoBeforeAfterValve.java:47)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.geronimo.tomcat.valve.ThreadCleanerValve.invoke(ThreadCleanerValve.java:40)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:736)
This is the content of the workflow.xml file:

<workflow-app xmlns="uri:oozie:workflow:0.1" name="ssh-wf">
<start to="SSHJob"/>
<action name="SSHJob">
<ssh>
<host>biadmin@bigdata</host>
<command>/tmp/testingestor/start-ingestor.sh</command>
<args>--config-file /tmp/testingestor/basic.conf</args>
<capture-output/>
</ssh>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>SSH job failed, error message${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="end"/>
</workflow-app>

Any help in order to solve this problem is appreciated.

Manuel Romo
Updated on 2012-12-07T03:37:27Z at 2012-12-07T03:37:27Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    603 Posts
    ACCEPTED ANSWER

    Re: Oozie workflow error (stream exceeds limit [2048]

    ‏2012-12-07T03:36:24Z  in response to SystemAdmin
    Hi Manuel,

    From the error output, I guess the output of your shell script is kind of large and exceeding 2048 bytes, which causes the issue. To resolve it, please try below steps:

    1. stop oozie service by command "stop.sh oozie"
    2. add below property in $BIGINSIGHTS_HOME/oozie/conf/oozie-site.xml(-1 means no limit)
    <property>
    <name>oozie.servlet.CallbackServlet.max.data.len</name>
    <value>-1</value>
    </property>
    3. restart oozie service by command "star.sh oozie"
    4. re-run your application

    Hope this could help, and let me know if you still see any issue.
    • SystemAdmin
      SystemAdmin
      603 Posts
      ACCEPTED ANSWER

      Re: Oozie workflow error (stream exceeds limit [2048]

      ‏2012-12-07T03:37:27Z  in response to SystemAdmin
      sorry for the typo in step #3, the command should be "start.sh oozie"