Topic
6 replies Latest Post - ‏2013-09-12T21:02:42Z by kutzki
kutzki
kutzki
5 Posts
ACCEPTED ANSWER

Pinned topic BI 2.1.0.0 Can't upload files into hadoop via files console

‏2013-09-11T20:16:27Z |

Hello... I've installed the 2.1.0.0 Version of BigInsight. I can't upload any files into the hadoop file tree subdirectories. I can create subdirectories, i can upload files to the root directory and i can even THEN move the file to the target subdirectory via GUI. i can set permissions and delet files. All no problem. But direct upload to a subdirectory isn't possible...

Does anyone has this problem also ? Are there any security settings ?

Installation is on RedHat 6.4 (4 Node Test cluster). Installation was fine without errors.

Updated on 2013-09-11T20:16:42Z at 2013-09-11T20:16:42Z by kutzki
  • DiLi
    DiLi
    69 Posts
    ACCEPTED ANSWER

    Re: BI 2.1.0.0 Can't upload files into hadoop via files console

    ‏2013-09-11T20:29:12Z  in response to kutzki

    Hello,
     

    Is the BI Console secured (requires user/pw to log in the Console UI) or non-secured?

    Are there any exceptions for the file upload? BI 2.1 uses a component called HttpFS to save file to HDFS. what you can try, to pinpoint the issue, is to leaving just one HttpFS running (you can start/stop individual HttpFS from the ClusterStatus tab), then try the file upload, while checking the HttpFS log at $BIGINSIGHTS_VAR/httpfs/logs/httpfs-audit.log on the server where HttpFS is running.

    Also check $BIGINSIGHTS_VAR/console/log/console.log for any errors thrown by the console during the file upload.

    • kutzki
      kutzki
      5 Posts
      ACCEPTED ANSWER

      Re: BI 2.1.0.0 Can't upload files into hadoop via files console

      ‏2013-09-11T20:41:07Z  in response to DiLi

      Hello DiLi,

      thanks for the fast help. I'll check it out and come back. The BI was installed with PAM Authentication.

      • kutzki
        kutzki
        5 Posts
        ACCEPTED ANSWER

        Re: BI 2.1.0.0 Can't upload files into hadoop via files console

        ‏2013-09-11T21:35:20Z  in response to kutzki

        Hello Dili,

        in the httpfs-audit.log is nothing to find on the "left" httpfs node (stopped 2 of 3). This log is empty. But in the console.log on the main node there is a difference during file upload. When i choose the root directory -> upload works and in the console.log a DataManagementAction entry shows that the fstype is HDFS. When i upload to a subdirectory the fstype is NULL (which surely is the reason why it didn't works). But i can't influence the file type by th GUI upload...

        Updated on 2013-09-11T21:35:59Z at 2013-09-11T21:35:59Z by kutzki
        • DiLi
          DiLi
          69 Posts
          ACCEPTED ANSWER

          Re: BI 2.1.0.0 Can't upload files into hadoop via files console

          ‏2013-09-11T21:41:45Z  in response to kutzki

          Hello,

          Can you run $BIGINSIGHTS_HOME/bin/biginsightslevel as the biadmin user and paste the BigInsights version and build level?

          Also, can you provide detailed reproduce steps? including which dir were you trying to upload file to, that's the permission of that dir, which user do you use to log in the BI Console, etc, more information the better.

          • kutzki
            kutzki
            5 Posts
            ACCEPTED ANSWER

            Re: BI 2.1.0.0 Can't upload files into hadoop via files console

            ‏2013-09-11T22:15:06Z  in response to DiLi

            Hello

            the output of biginsightslevel is

            IBM InfoSphere BigInsights Enterprise Edition v2.1.0.0, build linux64_b20130531_1113, type Enterprise

            I log into the BI Web Console with the biadmin (super) user. Go the "Files" tab. In the Navigation tree of HDFS i select the root directory and press "Create Folder" button. e.g. i create a folder "test". Then i click that Folder to highlight it and press the button "Upload...". I select the file from my local disk structure and press ok. Then : Nothing happens

            The permissions of the "test" dir is rwxr-wr-w (same as the root dir)

            In the console-web.log i find

            2013-09-12 00:08:57,894 INFO  [DataManagementAction] Data management...
            2013-09-12 00:08:57,894 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> fsType:null
            2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> action:uploade_progress_check
            2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > action:uploade_progress_check
            2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > fsType:null
            2013-09-12 00:08:57,905 INFO  [FileSystemHelper] FileSystemHelper > fs...DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-326547854_61, ugi=biadmin]]
             

            While when i upload to the root directory (which works)

            2013-09-12 00:10:59,527 INFO  [DataManagementAction] Data management...
            2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> fsType:hdfs
            2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> action:showfile
            2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > action:showfile
            2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > fsType:hdfs
            2013-09-12 00:10:59,538 INFO  [FileSystemHelper] FileSystemHelper > fs...DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-326547854_61, ugi=biadmin]]

            greetings
             

            • kutzki
              kutzki
              5 Posts
              ACCEPTED ANSWER

              Re: BI 2.1.0.0 Can't upload files into hadoop via files console

              ‏2013-09-12T21:02:42Z  in response to kutzki

              Got it -> Simple as always... Name resolution was the problem... The Data Nodes must be reachable by hostname... Simple error -> Big Trouble