Topic
  • 2 replies
  • Latest Post - ‏2013-02-25T09:30:44Z by EmiliePOCCIOLI
EmiliePOCCIOLI
EmiliePOCCIOLI
2 Posts

Pinned topic Error Injecting data to BigInsights

‏2013-02-19T11:06:37Z |
Hi,

I would like to inject data from Streams 3.0 to BigInsights 2.0 using HDFSFileSink operator.
Streams and BigInsights are installed on different VMs.
Here is a basic example I tried to test the connection but it fails:
namespace application;
use com.ibm.streams.bigdata.hdfs::*;

composite BigInsights {

graph
stream<rstring astring> stringSource = Beacon() {
logic state : {
mutable uint64 n = 0;
}
param
period: 0.01;
iterations : 101u ;
output
stringSource : astring = "hello";
}

() as BIOutput = HDFSFileSink(stringSource) {
param
format: txt;
useVersionOneApi : false;
file : "/video_streaming/out%FILENUM%.txt";
hdfsConfigFile: "./hdfsconfig.txt";

}

}

Here is the properties in the hdfsconfig.txt I'm using :
hdfsport=9000
hdfshost=bigdata.ibm.com
hdfsuser=biadmin
hdfsgroup=supergroup

And here is the error log I got:

Exception in thread "main" java.io.IOException: Call to bigdata.ibm.com/192.168.64.2:9000 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:386)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
19 Feb 2013 10:41:23.468 21162 ERROR #splapplog,J[0],P[0],BIOutput,HdfsCommon MBIOutput.cpp:getHdfsPtr:306 - Could not access HDFS file system on host bigdata.ibm.com on port 9,000.
19 Feb 2013 10:41:23.470 21162 ERROR #splapplog,J[0],P[0],BIOutput,spl_pe MPEImpl.cpp:logTerminatingException:1172 - CDISR5033E: An exception occurred during the execution of the BIOutput operator. Processing element number 0 is terminating.
19 Feb 2013 10:41:23.470 21162 ERROR #splapptrc,J[0],P[0],BIOutput,spl_operator MPEImpl.cpp:instantiateOperators:422 - CDISR5030E: An exception occurred during the execution of the BIOutput operator. The exception is Could not connect to HDFS.
19 Feb 2013 10:41:23.471 21162 ERROR #splapptrc,J[0],P[0],BIOutput,spl_pe MPEImpl.cpp:process:633 - CDISR5079E: An exception occurred during the processing of the processing element. The error is: Could not connect to HDFS.
19 Feb 2013 10:41:23.471 21162 ERROR #splapptrc,J[0],P[0],BIOutput,spl_operator MPEImpl.cpp:process:654 - CDISR5053E: Runtime failures occurred in the following operators: BIOutput.

All firewalls are disabled, I can telnet my BI VM on the port 9000 from the Streams VM.
I set the proper permissions to my /video_streaming directory for the user biadmin (group supergroup)
But no matter I put hdfsuser and hdfsgroup attributes in the hdfsconfig file, I always get the same error.

Any help would be really appreciated

Thanks in advance
Emilie
  • Stan
    Stan
    76 Posts

    Re: Error Injecting data to BigInsights

    ‏2013-02-21T00:50:35Z  
    I've not done this myself but noticed that the IP in the message is a private network IP. Could you name resolution be picking up the wrong IP?

    java.io.IOException: Call to bigdata.ibm.com/192.168.64.2:9000
  • EmiliePOCCIOLI
    EmiliePOCCIOLI
    2 Posts

    Re: Error Injecting data to BigInsights

    ‏2013-02-25T09:30:44Z  
    • Stan
    • ‏2013-02-21T00:50:35Z
    I've not done this myself but noticed that the IP in the message is a private network IP. Could you name resolution be picking up the wrong IP?

    java.io.IOException: Call to bigdata.ibm.com/192.168.64.2:9000
    Hi,
    I managed to fix the problem. The HADOOP_HOME variable was pointing to an old IHC directory.
    Thanks anyway