Topic
7 replies Latest Post - ‏2013-01-28T11:09:47Z by SystemAdmin
SystemAdmin
SystemAdmin
603 Posts
ACCEPTED ANSWER

Pinned topic "Running application..." never close, application never runs

‏2013-01-09T12:01:13Z |
I'm running BigInsights 2.0 under Centos 5 inside a VM.
When I try to run an application, which previously worked with BI 1.4, I get the attached message "Running application..." and it never goes away. So, the application is not launched.
If I refresh the page or open another one I can continue to use BI console, but this happen again when I start it again.
I tried to restart the platform and even the computer, but it still happens. It also happens with some sample apps, like Cluster Aggregation.
How can I know why this is happening and how to solve it?

Thanks,
Jacopo
Updated on 2013-01-28T11:09:47Z at 2013-01-28T11:09:47Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    603 Posts
    ACCEPTED ANSWER

    Re: "Running application..." never close, application never runs

    ‏2013-01-09T13:58:57Z  in response to SystemAdmin
    I forgot to mention that when it happens all applications start to behave this way, even simple ones like WordCounter, and cannot be run until I restart BigInsights.
    • SystemAdmin
      SystemAdmin
      603 Posts
      ACCEPTED ANSWER

      Re: "Running application..." never close, application never runs

      ‏2013-01-23T16:30:21Z  in response to SystemAdmin
      I still have the same problem, it just can't run my application and stay forever in that state.
      I tried to run it using the REST interface, but can't figure out the structure of the XML code to pass it.

      My application need only one argument, a file path, but can't figure out how to call it.

      If I call the URL
      http://localhost:8080/data/controller/ApplicationManagement?actiontype=run_application&runconfig=<runconfig><name>esecuzione di prova</name><appid>4ed3ee4f-678b-4a79-a35f-8d1fcacd11a0</appid><properties><property><name>inputDir</name><value paramtype='DIRECTORYPATH'>/user/hadoop/gutenberg</value></property></properties></runconfig>;

      I get the JSON errror response:
      {
      "result": {
      "errormessage": "java.lang.NullPointerException",
      "stacktrace": "java.lang.NullPointerException\n\tat com.ibm.biginsights.web.applicationmanagement.beans.AppPalette.runOozieJob(AppPalette.java:2036)\n\tat com.ibm.biginsights.web.applicationmanagement.action.ApplicationManagementAction.perform(ApplicationManagementAction.java:155)\n\tat com.ibm.biginsights.webapp.actions.Action.doAction(Action.java:62)\n\tat com.ibm.biginsights.webapp.ControllerServlet.helper(ControllerServlet.java:49)\n\tat com.ibm.biginsights.webapp.ControllerServlet.doGet(ControllerServlet.java:78)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:575)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:668)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1221)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:757)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:440)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1040)\n\tat com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:81)\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:930)\n\tat com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$1.run(DynamicVirtualHost.java:253)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:457)\n\tat com.ibm.ws.threading.internal.Worker.executeWork(Worker.java:398)\n\tat com.ibm.ws.threading.internal.Worker.run(Worker.java:380)\n\tat java.lang.Thread.run(Thread.java:738)\n",
      "status": "FAILED"
      }
      }

      But it doesn't help me to figure what is wrong.
      • SystemAdmin
        SystemAdmin
        603 Posts
        ACCEPTED ANSWER

        Re: "Running application..." never close, application never runs

        ‏2013-01-23T19:04:51Z  in response to SystemAdmin
        Hi JacopoF,

        Can you verify a couple of things?

        1) Make sure the application that you are trying to run is deployed first.
        2) Is the path a valid path?

        Can you paste the oozie log here (/var/ibm/biginsights/oozie/logs/oozie.log)

        Thanks,
        Harish
        • SystemAdmin
          SystemAdmin
          603 Posts
          ACCEPTED ANSWER

          Re: "Running application..." never close, application never runs

          ‏2013-01-24T09:39:59Z  in response to SystemAdmin
          > {quote:title=HarishKrishnan wrote:}{quote}
          > Hi JacopoF,
          >
          > Can you verify a couple of things?
          >
          > 1) Make sure the application that you are trying to run is deployed first.
          > 2) Is the path a valid path?
          >
          > Can you paste the oozie log here (/var/ibm/biginsights/oozie/logs/oozie.log)
          >
          > Thanks,
          > Harish
          Yes, the application is deployed, I see it in the Execute tab of the web console. The path is valid since I copy-pasted it from the console file explorer. I can't select it from the application page because the file selector looks like shown in the attached picture.
          The oozie log is attached, too, but I can't see anything after January, 21st 2013 in it, as well as in other logs in the oozie logs folder, though I run it right yesterday and today.

          The problem happens with every application, even the provided "Data sampling" one, while I ran it some weeks ago on the same machine.

          Cheers,
          Jacopo
          • SystemAdmin
            SystemAdmin
            603 Posts
            ACCEPTED ANSWER

            Re: "Running application..." never close, application never runs

            ‏2013-01-25T17:20:35Z  in response to SystemAdmin
            Looks like health check failed for oozie. Can you run /opt/ibm/biginsights/bin/status.sh hadoop oozie and check the status of both hadoop and oozie?
            If everything looks good, try a restart --- do a stop-all (/opt/ibm/biginsights/bin/stop-all.sh) and start-all (/opt/ibm/biginsights/bin/start-all.sh) because there is nothing that I could get from the log because they all are old logs.
            • SystemAdmin
              SystemAdmin
              603 Posts
              ACCEPTED ANSWER

              Re: "Running application..." never close, application never runs

              ‏2013-01-28T09:00:27Z  in response to SystemAdmin
              Did it, run stop-all.sh and start-all.sh, after this the output of status.sh is:

              INFO Progress - Status hadoop
              INFO @abardone01.replynet.prv - namenode started, pid 4016
              INFO @abardone01.replynet.prv - secondarynamenode started, pid 4299
              INFO @abardone01.replynet.prv - datanode started, pid 4461
              INFO Progress - 25%
              INFO @abardone01.replynet.prv - jobtracker started, pid 5139
              INFO @abardone01.replynet.prv - tasktracker started, pid 5300
              INFO Progress - 50%
              INFO Progress - Status oozie
              INFO @abardone01.replynet.prv - oozie started, pid 7595
              INFO Progress - 100%
              INFO DeployManager - Status; SUCCEEDED components: hadoop, oozie; Consumes : 3101ms
              but when I run the application, it still stay in that state and doesn't work, while status.sh give the same result.

              But this time I noticed a file, /var/ibm/biginsights/oozie/logs/messages_13.01.28_09.23.01.0.log, attached to this post.
              It seems to complain about a derby jar not found, my application used a derby in the past for an experiment, so I checked that the jar wasn't referenced in Eclipse, and it wasn't.
              I restarted the server (it didn't allow to undeploy the aplication), undeployed and deleted the application and reloaded it.
              Now it starts, then it gives an error which is probably a file not found on the application side, and I'll debug it.

              Is there a "collector" of log files or message errors to see all of them together ?

              Thanks for the help,
              Jacopo
            • SystemAdmin
              SystemAdmin
              603 Posts
              ACCEPTED ANSWER

              Re: "Running application..." never close, application never runs

              ‏2013-01-28T11:09:47Z  in response to SystemAdmin
              Ok. now it starts but give a new error.
              I run stop-all.sh and start-all.sh, after this the output of status.sh is:

              INFO Progress - Status hadoop
              INFO @peruser.intranet.prv - namenode started, pid 4016
              INFO @peruser.intranet.prv - secondarynamenode started, pid 4299
              INFO @peruser.intranet.prv - datanode started, pid 4461
              INFO Progress - 25%
              INFO @peruser.intranet.prv - jobtracker started, pid 5139
              INFO @peruser.intranet.prv - tasktracker started, pid 5300
              INFO Progress - 50%
              INFO Progress - Status oozie
              INFO @peruser.intranet.prv - oozie started, pid 7595
              INFO Progress - 100%
              INFO DeployManager - Status; SUCCEEDED components: hadoop, oozie; Consumes : 3101ms

              but when I run the application, it still stay in that state and doesn't work, while status.sh give the same result.

              But this time I noticed a file, /var/ibm/biginsights/oozie/logs/messages_13.01.28_09.23.01.0.log, attached to this post.
              It seems to complain about a derby jar not found, my application used a derby in the past for an experiment, so I checked that the jar wasn't referenced in Eclipse, and it wasn't.
              I restarted the server (it didn't allow to undeploy the aplication), undeployed and deleted the application and reloaded it.
              Now it starts, read my configuration file from the DFS, and tries to start a MapReduce job with the expected parameters.
              Then it fails, and from the log I read:

              2013-01-28 11:24:19,986 INFO com.project.JSONMR.EsecuzioneWorkflowJSON: preparing to run with args: [/user/applications/datasets/estratti.txt, /user/applications/datasets/results/classificati.json]
              2013-01-28 11:24:19,996 INFO com.project.JSONMR.EsecuzioneWorkflowJSON: job ready, about to start it...
              2013-01-28 11:24:20,121 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name jvm already exists!
              2013-01-28 11:24:20,137 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:biadmin cause:ENOENT: No such file or directory
              2013-01-28 11:24:20,239 INFO org.apache.hadoop.mapred.Task: Task:attempt_201301281112_0001_m_000000_0 is done. And is in the process of commiting
              2013-01-28 11:24:21,298 INFO org.apache.hadoop.mapred.Task: Task attempt_201301281112_0001_m_000000_0 is allowed to commit now
              2013-01-28 11:24:21,314 INFO org.apache.hadoop.mapred.FileOutputCommitter: Saved output of task 'attempt_201301281112_0001_m_000000_0' to hdfs://peruser.intranet.prv:9000/user/biadmin/oozie-biad/0000000-130128112056846-oozie-biad-W/action1--java/output
              2013-01-28 11:24:21,323 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201301281112_0001_m_000000_0' done.
              2013-01-28 11:24:21,354 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
              2013-01-28 11:24:21,395 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
              2013-01-28 11:24:21,396 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName biadmin for UID 500 from the native implementation

              while the exception is:
              Failing Oozie Launcher, Main class http://com.project.JSONMR.EsecutoreGenerico, main() threw exception, Wrapped ENOENT: No such file or directory (workflow#16)
              org.mozilla.javascript.WrappedException: Wrapped ENOENT: No such file or directory (workflow#16)
              at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Context.java:1754)
              at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:148)
              at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:225)
              at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
              at org.mozilla.javascript.gen.workflow_1._c_workflow_0(workflow:16)
              at org.mozilla.javascript.gen.workflow_1.call(workflow)
              at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
              at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3090)
              at org.mozilla.javascript.gen.workflow_1.call(workflow)
              at com.project.JSONMR.EsecutoreGenerico.main(EsecutoreGenerico.java:66)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
              at java.lang.reflect.Method.invoke(Method.java:611)
              at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
              at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
              at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
              at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
              at java.security.AccessController.doPrivileged(AccessController.java:310)
              at javax.security.auth.Subject.doAs(Subject.java:573)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1144)
              at org.apache.hadoop.mapred.Child.main(Child.java:249)
              Caused by: ENOENT: No such file or directory
              at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
              at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:653)
              at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:646)
              at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
              at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
              at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
              at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
              at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
              at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
              at java.security.AccessController.doPrivileged(AccessController.java:310)
              at javax.security.auth.Subject.doAs(Subject.java:573)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1144)
              at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
              at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
              at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1261)
              at com.project.JSONMR.EsecuzioneWorkflowJSON.run(EsecuzioneWorkflowJSON.java:207)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
              at com.project.JSONMR.EsecutoreHadoop.eseguiFase(EsecutoreHadoop.java:33)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
              at java.lang.reflect.Method.invoke(Method.java:611)
              at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
              ... 21 more

              From the web console I can see those files.

              Googling it, I found that ENOENT is something related to IBM DFS and on Stackoverflow I some people suggest to format HDFS for a similar error, but if I give the format command it will lost all the BigInsights data I see in the , I suppose

              Is there a "collector" of log files or message errors to see all of them together ?

              Thanks for the help,
              Jacopo