Topic
  • 28 replies
  • Latest Post - ‏2018-06-18T18:09:38Z by Nithish Shetty
Nithish Shetty
Nithish Shetty
16 Posts

Pinned topic Integrating ODPP libraries in Hadoop

‏2018-05-21T21:28:58Z | #bigdata #odpp_hadoop
Hi,
I am encountering issues while running the map-reduce program to mask the data in HDFS. Built the jar by integrating ODPP and Hadoop libraries in Eclipse along with map-reduce functions.
More details  -
Here is how I am trying to invoke the jar file in IBM BigData environment:
yarn jar /root/Downloads/ODPP/MapR_ODPP/java_test_odpp.jar -classpath "$ODPPLIBL/ODPPJava.jar" /tmp/ccn.txt /tmp/ccn_out_d1
[All the environment variables are set]
Input file (ccn.txt) looks like :
[root@rvm MapR_ODPP]# hadoop fs -cat /tmp/ccn.txt
CN: 4571444025760464
The output of the map-reduce job has to mask the sample credit card number. But unfortunately, I end up getting the 'ClassDefNotFound' error:
Exception in thread "main" java.lang.NoClassDefFoundError: com/ibm/nex/odpp/ODPP
        at MROdpp.main(MROdpp.java:101)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: com.ibm.nex.odpp.ODPP
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Appreciate if someone can assist me on this. Also, please let me know if in case you need any further details.
 
  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-05-22T10:32:14Z  

    Hi,

    I have tried this test case long back, that time it worked very well.

    I have attached required files, can you please try with these files and let me know how it goes.

     

    Thanks,

    Tulasi

    Attachments

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-05-22T14:29:49Z  

    Hi,

    I have tried this test case long back, that time it worked very well.

    I have attached required files, can you please try with these files and let me know how it goes.

     

    Thanks,

    Tulasi

    Thank you very much for your quick response on this, Tulasi. I'll try that out and let you know.

    Regards,

    Nithish

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-05-22T20:29:23Z  

    Hi,

    I have tried this test case long back, that time it worked very well.

    I have attached required files, can you please try with these files and let me know how it goes.

     

    Thanks,

    Tulasi

    Hi Tulasi,

    Actually, my question is based on map reduce functionality by leveraging the JAVA API provided in the FP05. When I invoke the jar file that was built out of my Map-Reduce code, I end up getting 'ClassDefNotFound' error. Any insights on this please?

  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-05-24T09:02:00Z  

    Hi Tulasi,

    Actually, my question is based on map reduce functionality by leveraging the JAVA API provided in the FP05. When I invoke the jar file that was built out of my Map-Reduce code, I end up getting 'ClassDefNotFound' error. Any insights on this please?

    Hi Nitish,

    I am checking what are trying and I will get back to you as and when I have something to share with you.

    Thanks,

    Tulasi

  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-05-26T11:48:03Z  

    Hi Nitish,

    I am checking what are trying and I will get back to you as and when I have something to share with you.

    Thanks,

    Tulasi

    Hi Nitish,

     

    I am trying to understand the steps you followed. Did you install ODPP on IBM BigInsighits environment? You modified the sample ODPPJavaSample and tried to run and that is failing?

     

    Thanks,

    Tulasi

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-05-30T16:59:52Z  

    Hi Nitish,

     

    I am trying to understand the steps you followed. Did you install ODPP on IBM BigInsighits environment? You modified the sample ODPPJavaSample and tried to run and that is failing?

     

    Thanks,

    Tulasi

    Hi Tulasi, 

    That's correct. I have installed ODPP on IBM Big Insights. Rewrote Java Sample to align with Map Reduce Paradigm. 

    Please let me know if need further information.

    Thank you!

  • PeterCostigan
    PeterCostigan
    20 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-05-30T22:22:43Z  

    Hi Nitish, now I'm really curious too-- what do you mean by rewriting the Java Sample "to align with Map Reduce Paradigm"?

     

    Thanks,

    Peter

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-05-31T14:51:23Z  

    Hi Nitish, now I'm really curious too-- what do you mean by rewriting the Java Sample "to align with Map Reduce Paradigm"?

     

    Thanks,

    Peter

    Hi Peter,

    Here is what I meant - Basically, I reused the libraries (ODPP) that were used by Java Sample(provided in FP05) in my map-reduce program in java. Here is the logic - mapper would map key-value pairs from the file on HDFS. Reducer would reduce it to the output. Please let me know if you need further clarifications.

     

    -Nithish 

  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-04T04:26:56Z  

    Hi Peter,

    Here is what I meant - Basically, I reused the libraries (ODPP) that were used by Java Sample(provided in FP05) in my map-reduce program in java. Here is the logic - mapper would map key-value pairs from the file on HDFS. Reducer would reduce it to the output. Please let me know if you need further clarifications.

     

    -Nithish 

    Hi Nitish,

     

    I have setup ODPP on my Hadoop environment. Is it possible to share your java sample code,I will check and get back to you.

     

    Thanks,

    Tulasi

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-04T15:29:11Z  

    Hi Nitish,

     

    I have setup ODPP on my Hadoop environment. Is it possible to share your java sample code,I will check and get back to you.

     

    Thanks,

    Tulasi

    Hi Tulasi,

     

    Please find the attached folders. Kindly let me know if this would suffice.

     

    Thank you,

    Nithish

    Attachments

    Updated on 2018-06-04T15:29:45Z at 2018-06-04T15:29:45Z by Nithish Shetty
  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-06T07:16:50Z  

    Hi Tulasi,

     

    Please find the attached folders. Kindly let me know if this would suffice.

     

    Thank you,

    Nithish

    Hi Nitish,

    I guess the purpose of ODPPJava.jar is different.

    Can you try with mod/MODApp/ioqmoda.jar

    yarn/hadoop jar ioqmoda.jar -i <inputfile> --hadoop used to do this job but when I checked it is failing. I trying to fix this.

     

    Thanks,

    Tulasi

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-06T15:08:28Z  

    Hi Nitish,

    I guess the purpose of ODPPJava.jar is different.

    Can you try with mod/MODApp/ioqmoda.jar

    yarn/hadoop jar ioqmoda.jar -i <inputfile> --hadoop used to do this job but when I checked it is failing. I trying to fix this.

     

    Thanks,

    Tulasi

    Hi Tulasi,

    I had tried ioqmoda.jar a month ago. But unfortunately, that didn't seem to work either.

    Attempted again for your reference -

     

    [root@rvm MapR_ODPP]# ll ioqmoda.jar
    -rw-r--r--. 1 root root 7602599 May 22 07:36 ioqmoda.jar
    [root@rvm MapR_ODPP]# yarn ioqmoda.jar -i ccn.txt
    Error: Could not find or load main class ioqmoda.jar
    [root@rvm MapR_ODPP]# hadoop ioqmoda.jar -i ccn.txt
    Error: Could not find or load main class ioqmoda.jar
     

    Updated on 2018-06-06T15:09:09Z at 2018-06-06T15:09:09Z by Nithish Shetty
  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-06T16:08:20Z  

    Hi Tulasi,

    I had tried ioqmoda.jar a month ago. But unfortunately, that didn't seem to work either.

    Attempted again for your reference -

     

    [root@rvm MapR_ODPP]# ll ioqmoda.jar
    -rw-r--r--. 1 root root 7602599 May 22 07:36 ioqmoda.jar
    [root@rvm MapR_ODPP]# yarn ioqmoda.jar -i ccn.txt
    Error: Could not find or load main class ioqmoda.jar
    [root@rvm MapR_ODPP]# hadoop ioqmoda.jar -i ccn.txt
    Error: Could not find or load main class ioqmoda.jar
     

    That way it will not work. Check this document https://www.ibm.com/support/knowledgecenter/en/SSMLQ4_9.1.0/com.ibm.nex.mod.doc/topics/opmod-r-command_line_syntax.html

     

    It worked with Optim 9.1 but with 11.3 it is failing. I am fixing this and it will take some time.

    Is there any PMR opened for this issue?

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-06T16:15:27Z  

    That way it will not work. Check this document https://www.ibm.com/support/knowledgecenter/en/SSMLQ4_9.1.0/com.ibm.nex.mod.doc/topics/opmod-r-command_line_syntax.html

     

    It worked with Optim 9.1 but with 11.3 it is failing. I am fixing this and it will take some time.

    Is there any PMR opened for this issue?

    Sure. Thanks Tulasi.

    And coming to PMR, yes we have logged one for this issue.

  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-06T16:17:53Z  

    Sure. Thanks Tulasi.

    And coming to PMR, yes we have logged one for this issue.

    Can you please let me know PMR number, this helps me to track properly. Thanks.
     

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-06T16:45:56Z  

    Can you please let me know PMR number, this helps me to track properly. Thanks.
     

    Here you go - 
    PMR 42458,756,000

  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-07T04:49:12Z  

    Here you go - 
    PMR 42458,756,000

    Hi Nitish,

     

    Try with the attached ioqmoda.jar

    Inorder to execute this, change the last line of IOQMaskingTool.sh to "yarn or hadoop jar ./ioqmoda.jar -i <input configuration file> -h"

     

    Thanks,

    Tulasi

    Attachments

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-07T20:15:39Z  

    Hi Nitish,

     

    Try with the attached ioqmoda.jar

    Inorder to execute this, change the last line of IOQMaskingTool.sh to "yarn or hadoop jar ./ioqmoda.jar -i <input configuration file> -h"

     

    Thanks,

    Tulasi

    Hi Tulasi,

     

    MODApp works fine but my issue is more oriented with the custom API for the map-reduce functionality. As you know, we can not modify the inbuilt ioqmoda jar based on our requirements. Correct? The reason why I have built the custom jar out of the map-reduce program is that I could tweak the program and run the modified jar file on HDFS as and when the requirements keep changing.  JAVA API  in the FP05 is the cornerstone of this work.  Please let me know if in case you need further clarification on this.

     

    Thank you,

     

    Nithish

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-07T20:33:30Z  

    Hi Nitish,

    I guess the purpose of ODPPJava.jar is different.

    Can you try with mod/MODApp/ioqmoda.jar

    yarn/hadoop jar ioqmoda.jar -i <inputfile> --hadoop used to do this job but when I checked it is failing. I trying to fix this.

     

    Thanks,

    Tulasi

    Yes, Tulasi. I agree that the purpose of ODPPJava.jar is different. In our case, we are building a custom jar that can integrate hadoop and ODPP libraries. Therefore, we aren't reusing the ODPPJava.jar in our case.

  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-08T04:06:30Z  

    Yes, Tulasi. I agree that the purpose of ODPPJava.jar is different. In our case, we are building a custom jar that can integrate hadoop and ODPP libraries. Therefore, we aren't reusing the ODPPJava.jar in our case.

    Hi Nithish,

    Thanks for confirming that Map Reduce with ioqmoda works.

    Yes, I understand your problem and I am trying to troubleshoot your sample code. I will get back to you as soon as I have some progress.

    Thanks,

    Tulasi

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-08T15:23:45Z  

    Hi Nithish,

    Thanks for confirming that Map Reduce with ioqmoda works.

    Yes, I understand your problem and I am trying to troubleshoot your sample code. I will get back to you as soon as I have some progress.

    Thanks,

    Tulasi

    Sure. Thank you Tulasi!

  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-11T06:14:49Z  

    Sure. Thank you Tulasi!

    Hi Nitish,

    When I opened your Java program on my eclipse, it shows me getInstance is defined for the type Job.

    Can you change this line "Job job = Job.getInstance(conf, "ODPP");     //failing here"

    to    " Job job = new Job(conf, "ODPP");"

    Then try?

    Thanks,

    Tulasi

     

  • Nithish Shetty
    Nithish Shetty
    16 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-11T18:34:09Z  

    Hi Nitish,

    When I opened your Java program on my eclipse, it shows me getInstance is defined for the type Job.

    Can you change this line "Job job = Job.getInstance(conf, "ODPP");     //failing here"

    to    " Job job = new Job(conf, "ODPP");"

    Then try?

    Thanks,

    Tulasi

     

    Thanks Tulasi!

    I replaced it with - Job job = new Job(conf, "ODPP"); and built a jar out of the updated code. No issues until that point but when I run the job, I end up seeing the same error msg :( 

      Exception in thread "main" java.lang.NoClassDefFoundError: com/ibm/nex/odpp

     

     

  • Tulasi D Uppu
    Tulasi D Uppu
    239 Posts

    Re: Integrating ODPP libraries in Hadoop

    ‏2018-06-12T07:01:44Z  

    Thanks Tulasi!

    I replaced it with - Job job = new Job(conf, "ODPP"); and built a jar out of the updated code. No issues until that point but when I run the job, I end up seeing the same error msg :( 

      Exception in thread "main" java.lang.NoClassDefFoundError: com/ibm/nex/odpp

     

     

    For some reason, I am getting below error:

     

    [tulasi@optim-rhel72-uppu MODApp]$ hadoop jar ./MROdpp.jar -classpath ../ODPP/Images64/rhel/bin/ODPPJava.jar /home/tulasi/ccn.txt /home/tulasi/ccn_out
    Exception in thread "main" java.lang.ClassNotFoundException: -classpath
            at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
            at java.lang.Class.forName0(Native Method)
            at java.lang.Class.forName(Class.java:348)
            at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

     

    How you are creating your jar file? Can you attach your jar file. I have attached mine.

     

    Attachments