I have a project where I have added the big data toolkit (in dependencies) and an HDFS File Sink at the end to create my output in HDFS, and have encountered a strange problem I did not encounter in my previous project using the bigdata toolkit.
The project will compile fine (I have the bigdata toolkit installed, I have the bigdata toolkit in dependencies, I have a valid HADOOP_HOME set). Another project in the same workspace will compile and run fine in Standalone mode. When I try to run this particular project in "Standalone" mode, I get:
Exception in thread "Thread-11" java.lang.NoClassDefFoundError: org.apache.hadoop.conf.Configuration
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
Which does not happen for my other project (I can run the other project fine in Standalone mode). However, if I compile this project at the command line and run it distributed, it works fine. This project is different from the other in that it uses the com.ibm.streams.text toolkit (the other project uses the SPSS analytics toolkit, but not the Text toolkit). I even tried going into the 'Edit Configuration' from the Launch Dialog and hard-coding an additional env setting for HADOOP_HOME just to be even more explicit about the presence of the env variable.
This isn't super critical since I have a workaround that can keep me moving forward, but development and debugging would be easier if I could conveniently work in standalone mode.
This (and the workaround of using the distributed deployment rather than standalone deployment) was discussed towards the end of the thread: https://www.ibm.com/developerworks/community/forums/html/topic?id=d428aa44-06bb-455a-85f6-172baa3d4e33&ps=25, but there wasn't a mention of a way to get this working in standalone mode (which is really great while in the dev/debugging process), so I thought I'd ask in case anybody knows the answer. The theory in this thread was along the lines that there was a problem with two java based toolkits somehow conflicting (?), but there wasn't anything certain or definate.
Thanks in advance for any help or assistance you can provide!