Topic
  • 7 replies
  • Latest Post - ‏2014-12-10T00:52:17Z by GJSissons
7PJN_Martin_Mašek
7PJN_Martin_Mašek
17 Posts

Pinned topic BigInsignts with IBM Platform Symphony

‏2014-08-11T20:56:03Z |

Hello everyone.

I've already made couple of posts around installation of BigInsights. Our client wants everything apart from classic BI distribution (they want GPFS, Adaptive map reduce and Platform Symphony). So I'm trying to install Platform symphony and integrate it with BI EE 3.0. I found this guide http://www.redbooks.ibm.com/redpapers/pdfs/redp5006.pdf but it is very depreciated. There is no reference about Platform symphony in installation guide of BigInsights 3.0 as there is no reference about BI in Platform symphony installation guide. I didn't find anything here, on the forum, either. So I think this topic could also serve as a reference for others.

I will try to understand the concepts of PS and try to figure out where to start. Meanwhile, I would really appreciate if someone was so kind as to give me a direction where to go or how to integrate PS with BI. I downloaded instructions about PS at http://w3-103.ibm.com/software/xl/download/ticket.do so I'm looking through it now.

  • mrmadira
    mrmadira
    26 Posts
    ACCEPTED ANSWER

    Re: BigInsignts with IBM Platform Symphony

    ‏2014-08-13T08:21:30Z  

    You will not find explicit references to PS in documentation/installation guides. Adaptive MR is the new name for PS.
    To install BI with Adaptive MR - read the section "Enabling adaptive MapReduce" in the installation guide.

    Note: Adaptive MapReduce is not included as part of the InfoSphere BigInsights
    Quick Start Edition.

    Updated on 2014-08-13T08:24:08Z at 2014-08-13T08:24:08Z by mrmadira
  • mrmadira
    mrmadira
    26 Posts

    Re: BigInsignts with IBM Platform Symphony

    ‏2014-08-13T08:21:30Z  

    You will not find explicit references to PS in documentation/installation guides. Adaptive MR is the new name for PS.
    To install BI with Adaptive MR - read the section "Enabling adaptive MapReduce" in the installation guide.

    Note: Adaptive MapReduce is not included as part of the InfoSphere BigInsights
    Quick Start Edition.

    Updated on 2014-08-13T08:24:08Z at 2014-08-13T08:24:08Z by mrmadira
  • 7PJN_Martin_Mašek
    7PJN_Martin_Mašek
    17 Posts

    Re: BigInsignts with IBM Platform Symphony

    ‏2014-08-14T07:40:32Z  
    • mrmadira
    • ‏2014-08-13T08:21:30Z

    You will not find explicit references to PS in documentation/installation guides. Adaptive MR is the new name for PS.
    To install BI with Adaptive MR - read the section "Enabling adaptive MapReduce" in the installation guide.

    Note: Adaptive MapReduce is not included as part of the InfoSphere BigInsights
    Quick Start Edition.

    Thx for clearing things up. I think it should be explicitly written somewhere.

  • GJSissons
    GJSissons
    5 Posts

    Re: BigInsignts with IBM Platform Symphony

    ‏2014-11-28T19:16:56Z  

    Hi folks - I was posting a question to the forum and just came across this question. I wrote a document explaining how to take advantage of Platform Symphony / Adaptive MapReduce with BigInsights. At the time I did this I was running a BigInsights 2.1 cluster to generate the examples, but I believe this is all still valid with BigInsights 3.x. I had hoped to turn this into a proper RedPaper but haven't had the time to pursue further. Hope this is helpful to someone.

  • yuefeng
    yuefeng
    7 Posts

    Re: BigInsignts with IBM Platform Symphony

    ‏2014-12-09T01:15:48Z  
    • GJSissons
    • ‏2014-11-28T19:16:56Z

    Hi folks - I was posting a question to the forum and just came across this question. I wrote a document explaining how to take advantage of Platform Symphony / Adaptive MapReduce with BigInsights. At the time I did this I was running a BigInsights 2.1 cluster to generate the examples, but I believe this is all still valid with BigInsights 3.x. I had hoped to turn this into a proper RedPaper but haven't had the time to pursue further. Hope this is helpful to someone.

    Hi GJ,

    thanks very much! It is OK when i add new application profile/consumer under MapReduceConsumer and separated user for above by following your guide . But i found that it was always running jobs under default consumer mapreduce611 application even i submit the job with new user . It didn't run under new application which i added . Hope to get your help .

  • GJSissons
    GJSissons
    5 Posts

    Re: BigInsignts with IBM Platform Symphony

    ‏2014-12-09T20:26:10Z  
    • yuefeng
    • ‏2014-12-09T01:15:48Z

    Hi GJ,

    thanks very much! It is OK when i add new application profile/consumer under MapReduceConsumer and separated user for above by following your guide . But i found that it was always running jobs under default consumer mapreduce611 application even i submit the job with new user . It didn't run under new application which i added . Hope to get your help .

    Hello youfeng,

    Sorry - this was not very well explained in my document, but you can over-ride the default setting in pmr-site.xml by specifying the name of the symphony application definition as a parameter on the command line.  Here is an example:

    hadoop jar ${HADOOP_HOME}/hadoop-example.jar sleep -Dmapreduce.application.name=Sqoop -m 2 -r 10 -mt 2000 -rt 2000

    The above would try and run the sleep command against an application definition called "Sqoop"

    This will over-ride the default application name in pmr-site.xml. As you say - by default the application name in pmr-site.xml points to the name of the MapReduce implementation for Platform Symphony bundled in BigInsights - I believe for BigInsights 3.0 this is Symphony 6.1.1 as you indicate. Hope this helps.

     

  • yuefeng
    yuefeng
    7 Posts

    Re: BigInsignts with IBM Platform Symphony

    ‏2014-12-10T00:33:14Z  
    • GJSissons
    • ‏2014-12-09T20:26:10Z

    Hello youfeng,

    Sorry - this was not very well explained in my document, but you can over-ride the default setting in pmr-site.xml by specifying the name of the symphony application definition as a parameter on the command line.  Here is an example:

    hadoop jar ${HADOOP_HOME}/hadoop-example.jar sleep -Dmapreduce.application.name=Sqoop -m 2 -r 10 -mt 2000 -rt 2000

    The above would try and run the sleep command against an application definition called "Sqoop"

    This will over-ride the default application name in pmr-site.xml. As you say - by default the application name in pmr-site.xml points to the name of the MapReduce implementation for Platform Symphony bundled in BigInsights - I believe for BigInsights 3.0 this is Symphony 6.1.1 as you indicate. Hope this helps.

     

    Hi GJ,

    I've tried with same result even submit job by running following command :

    hadoop jar ${HADOOP_HOME}/hadoop-example.jar sleep -Dmapreduce.application.name=Sqoop -m 2 -r 10 -mt 2000 -rt 2000

    The job still runs under MapReduce611 application other than the application i created .

  • GJSissons
    GJSissons
    5 Posts

    Re: BigInsignts with IBM Platform Symphony

    ‏2014-12-10T00:52:17Z  
    • yuefeng
    • ‏2014-12-10T00:33:14Z

    Hi GJ,

    I've tried with same result even submit job by running following command :

    hadoop jar ${HADOOP_HOME}/hadoop-example.jar sleep -Dmapreduce.application.name=Sqoop -m 2 -r 10 -mt 2000 -rt 2000

    The job still runs under MapReduce611 application other than the application i created .

    Thank you - I'm sorry Yuefeng. I'm at a loss to explain why this works for me and not for you. I assume that the additional applications you defined in Symphony were called "Sqoop" - Obviously the name "Sqoop" needs to match what was defined by you as an application name.  Not sure how to respond. I know this works for me, but devil in the details always.

    This is clearly complex and not something I can respond to via this chat - if you can please e-mail me out of band at gord@sissons.ca or gsissons@ca.ibm.com I can try and help you escalate this to our development team - Let's please escalate only if this is a customer requirement? - thank you very much. Sincerely appreciate your note - I cannot re-produce this problem and I am concerned I am missing something. Appreciate any details you can share.