Topic
  • 4 replies
  • Latest Post - ‏2012-10-09T17:36:53Z by VsV
VsV
VsV
10 Posts

Pinned topic Import from Nutch

‏2012-10-05T19:07:39Z |
Is it possible to import already crawled data(via Nutch) to the BigInsights collection(create BI collection from Nutch crawled data)?
Updated on 2012-10-09T17:36:53Z at 2012-10-09T17:36:53Z by VsV
  • SystemAdmin
    SystemAdmin
    603 Posts

    Re: Import from Nutch

    ‏2012-10-08T17:22:45Z  
    Hi,

    Have you thought of using something like 'distcp' or GUI version called 'Distributed Copy' to upload the file to HDFS:
    http://pic.dhe.ibm.com/infocenter/bigins/v1r4/topic/com.ibm.swg.im.infosphere.biginsights.dev.doc/doc/c_sample_apps_distcopy.html

    Once the data is present in HDFS, you should be able to create your collection(s).

    Please let me know in case if I misunderstood your question.

    Thanks,

    Zach
  • VsV
    VsV
    10 Posts

    Re: Import from Nutch

    ‏2012-10-08T19:51:12Z  
    Hi,

    Have you thought of using something like 'distcp' or GUI version called 'Distributed Copy' to upload the file to HDFS:
    http://pic.dhe.ibm.com/infocenter/bigins/v1r4/topic/com.ibm.swg.im.infosphere.biginsights.dev.doc/doc/c_sample_apps_distcopy.html

    Once the data is present in HDFS, you should be able to create your collection(s).

    Please let me know in case if I misunderstood your question.

    Thanks,

    Zach
    Thanks for your reply.

    Does the Reader exist in BigInsights for reading Nutch data and create a collection from them?

    Thank you!
  • SystemAdmin
    SystemAdmin
    603 Posts

    Re: Import from Nutch

    ‏2012-10-09T16:34:08Z  
    • VsV
    • ‏2012-10-08T19:51:12Z
    Thanks for your reply.

    Does the Reader exist in BigInsights for reading Nutch data and create a collection from them?

    Thank you!
    Hi,

    This is the answer I recieved from our dev -

    "As long as file system structure is same, you can use Base Crawl Data reader in bigsheets."

    Thanks,

    Zach
  • VsV
    VsV
    10 Posts

    Re: Import from Nutch

    ‏2012-10-09T17:36:53Z  
    Hi,

    This is the answer I recieved from our dev -

    "As long as file system structure is same, you can use Base Crawl Data reader in bigsheets."

    Thanks,

    Zach
    Thanks! It works.