Topic
  • 6 replies
  • Latest Post - ‏2013-09-12T21:02:42Z by kutzki
kutzki
kutzki
5 Posts

Pinned topic BI 2.1.0.0 Can't upload files into hadoop via files console

‏2013-09-11T20:16:27Z |

Hello... I've installed the 2.1.0.0 Version of BigInsight. I can't upload any files into the hadoop file tree subdirectories. I can create subdirectories, i can upload files to the root directory and i can even THEN move the file to the target subdirectory via GUI. i can set permissions and delet files. All no problem. But direct upload to a subdirectory isn't possible...

Does anyone has this problem also ? Are there any security settings ?

Installation is on RedHat 6.4 (4 Node Test cluster). Installation was fine without errors.

Updated on 2013-09-11T20:16:42Z at 2013-09-11T20:16:42Z by kutzki
  • kutzki
    kutzki
    5 Posts
    ACCEPTED ANSWER

    Re: BI 2.1.0.0 Can't upload files into hadoop via files console

    ‏2013-09-12T21:02:42Z  
    • kutzki
    • ‏2013-09-11T22:15:06Z

    Hello

    the output of biginsightslevel is

    IBM InfoSphere BigInsights Enterprise Edition v2.1.0.0, build linux64_b20130531_1113, type Enterprise

    I log into the BI Web Console with the biadmin (super) user. Go the "Files" tab. In the Navigation tree of HDFS i select the root directory and press "Create Folder" button. e.g. i create a folder "test". Then i click that Folder to highlight it and press the button "Upload...". I select the file from my local disk structure and press ok. Then : Nothing happens

    The permissions of the "test" dir is rwxr-wr-w (same as the root dir)

    In the console-web.log i find

    2013-09-12 00:08:57,894 INFO  [DataManagementAction] Data management...
    2013-09-12 00:08:57,894 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> fsType:null
    2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> action:uploade_progress_check
    2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > action:uploade_progress_check
    2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > fsType:null
    2013-09-12 00:08:57,905 INFO  [FileSystemHelper] FileSystemHelper > fs...DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-326547854_61, ugi=biadmin]]
     

    While when i upload to the root directory (which works)

    2013-09-12 00:10:59,527 INFO  [DataManagementAction] Data management...
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> fsType:hdfs
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> action:showfile
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > action:showfile
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > fsType:hdfs
    2013-09-12 00:10:59,538 INFO  [FileSystemHelper] FileSystemHelper > fs...DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-326547854_61, ugi=biadmin]]

    greetings
     

    Got it -> Simple as always... Name resolution was the problem... The Data Nodes must be reachable by hostname... Simple error -> Big Trouble

  • DiLi
    DiLi
    69 Posts

    Re: BI 2.1.0.0 Can't upload files into hadoop via files console

    ‏2013-09-11T20:29:12Z  

    Hello,
     

    Is the BI Console secured (requires user/pw to log in the Console UI) or non-secured?

    Are there any exceptions for the file upload? BI 2.1 uses a component called HttpFS to save file to HDFS. what you can try, to pinpoint the issue, is to leaving just one HttpFS running (you can start/stop individual HttpFS from the ClusterStatus tab), then try the file upload, while checking the HttpFS log at $BIGINSIGHTS_VAR/httpfs/logs/httpfs-audit.log on the server where HttpFS is running.

    Also check $BIGINSIGHTS_VAR/console/log/console.log for any errors thrown by the console during the file upload.

  • kutzki
    kutzki
    5 Posts

    Re: BI 2.1.0.0 Can't upload files into hadoop via files console

    ‏2013-09-11T20:41:07Z  
    • DiLi
    • ‏2013-09-11T20:29:12Z

    Hello,
     

    Is the BI Console secured (requires user/pw to log in the Console UI) or non-secured?

    Are there any exceptions for the file upload? BI 2.1 uses a component called HttpFS to save file to HDFS. what you can try, to pinpoint the issue, is to leaving just one HttpFS running (you can start/stop individual HttpFS from the ClusterStatus tab), then try the file upload, while checking the HttpFS log at $BIGINSIGHTS_VAR/httpfs/logs/httpfs-audit.log on the server where HttpFS is running.

    Also check $BIGINSIGHTS_VAR/console/log/console.log for any errors thrown by the console during the file upload.

    Hello DiLi,

    thanks for the fast help. I'll check it out and come back. The BI was installed with PAM Authentication.

  • kutzki
    kutzki
    5 Posts

    Re: BI 2.1.0.0 Can't upload files into hadoop via files console

    ‏2013-09-11T21:35:20Z  
    • kutzki
    • ‏2013-09-11T20:41:07Z

    Hello DiLi,

    thanks for the fast help. I'll check it out and come back. The BI was installed with PAM Authentication.

    Hello Dili,

    in the httpfs-audit.log is nothing to find on the "left" httpfs node (stopped 2 of 3). This log is empty. But in the console.log on the main node there is a difference during file upload. When i choose the root directory -> upload works and in the console.log a DataManagementAction entry shows that the fstype is HDFS. When i upload to a subdirectory the fstype is NULL (which surely is the reason why it didn't works). But i can't influence the file type by th GUI upload...

    Updated on 2013-09-11T21:35:59Z at 2013-09-11T21:35:59Z by kutzki
  • DiLi
    DiLi
    69 Posts

    Re: BI 2.1.0.0 Can't upload files into hadoop via files console

    ‏2013-09-11T21:41:45Z  
    • kutzki
    • ‏2013-09-11T21:35:20Z

    Hello Dili,

    in the httpfs-audit.log is nothing to find on the "left" httpfs node (stopped 2 of 3). This log is empty. But in the console.log on the main node there is a difference during file upload. When i choose the root directory -> upload works and in the console.log a DataManagementAction entry shows that the fstype is HDFS. When i upload to a subdirectory the fstype is NULL (which surely is the reason why it didn't works). But i can't influence the file type by th GUI upload...

    Hello,

    Can you run $BIGINSIGHTS_HOME/bin/biginsightslevel as the biadmin user and paste the BigInsights version and build level?

    Also, can you provide detailed reproduce steps? including which dir were you trying to upload file to, that's the permission of that dir, which user do you use to log in the BI Console, etc, more information the better.

  • kutzki
    kutzki
    5 Posts

    Re: BI 2.1.0.0 Can't upload files into hadoop via files console

    ‏2013-09-11T22:15:06Z  
    • DiLi
    • ‏2013-09-11T21:41:45Z

    Hello,

    Can you run $BIGINSIGHTS_HOME/bin/biginsightslevel as the biadmin user and paste the BigInsights version and build level?

    Also, can you provide detailed reproduce steps? including which dir were you trying to upload file to, that's the permission of that dir, which user do you use to log in the BI Console, etc, more information the better.

    Hello

    the output of biginsightslevel is

    IBM InfoSphere BigInsights Enterprise Edition v2.1.0.0, build linux64_b20130531_1113, type Enterprise

    I log into the BI Web Console with the biadmin (super) user. Go the "Files" tab. In the Navigation tree of HDFS i select the root directory and press "Create Folder" button. e.g. i create a folder "test". Then i click that Folder to highlight it and press the button "Upload...". I select the file from my local disk structure and press ok. Then : Nothing happens

    The permissions of the "test" dir is rwxr-wr-w (same as the root dir)

    In the console-web.log i find

    2013-09-12 00:08:57,894 INFO  [DataManagementAction] Data management...
    2013-09-12 00:08:57,894 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> fsType:null
    2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> action:uploade_progress_check
    2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > action:uploade_progress_check
    2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > fsType:null
    2013-09-12 00:08:57,905 INFO  [FileSystemHelper] FileSystemHelper > fs...DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-326547854_61, ugi=biadmin]]
     

    While when i upload to the root directory (which works)

    2013-09-12 00:10:59,527 INFO  [DataManagementAction] Data management...
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> fsType:hdfs
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> action:showfile
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > action:showfile
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > fsType:hdfs
    2013-09-12 00:10:59,538 INFO  [FileSystemHelper] FileSystemHelper > fs...DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-326547854_61, ugi=biadmin]]

    greetings
     

  • kutzki
    kutzki
    5 Posts

    Re: BI 2.1.0.0 Can't upload files into hadoop via files console

    ‏2013-09-12T21:02:42Z  
    • kutzki
    • ‏2013-09-11T22:15:06Z

    Hello

    the output of biginsightslevel is

    IBM InfoSphere BigInsights Enterprise Edition v2.1.0.0, build linux64_b20130531_1113, type Enterprise

    I log into the BI Web Console with the biadmin (super) user. Go the "Files" tab. In the Navigation tree of HDFS i select the root directory and press "Create Folder" button. e.g. i create a folder "test". Then i click that Folder to highlight it and press the button "Upload...". I select the file from my local disk structure and press ok. Then : Nothing happens

    The permissions of the "test" dir is rwxr-wr-w (same as the root dir)

    In the console-web.log i find

    2013-09-12 00:08:57,894 INFO  [DataManagementAction] Data management...
    2013-09-12 00:08:57,894 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> fsType:null
    2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> action:uploade_progress_check
    2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > action:uploade_progress_check
    2013-09-12 00:08:57,895 INFO  [DataManagementAction] DataManagementAction > fsType:null
    2013-09-12 00:08:57,905 INFO  [FileSystemHelper] FileSystemHelper > fs...DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-326547854_61, ugi=biadmin]]
     

    While when i upload to the root directory (which works)

    2013-09-12 00:10:59,527 INFO  [DataManagementAction] Data management...
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> fsType:hdfs
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > httpReq.getParameter >> action:showfile
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > action:showfile
    2013-09-12 00:10:59,527 INFO  [DataManagementAction] DataManagementAction > fsType:hdfs
    2013-09-12 00:10:59,538 INFO  [FileSystemHelper] FileSystemHelper > fs...DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-326547854_61, ugi=biadmin]]

    greetings
     

    Got it -> Simple as always... Name resolution was the problem... The Data Nodes must be reachable by hostname... Simple error -> Big Trouble