Topic
  • 4 replies
  • Latest Post - ‏2013-12-11T06:03:50Z by Debabrata.Ghosh
SystemAdmin
SystemAdmin
603 Posts

Pinned topic Gnip PowerTrack Import Error

‏2013-01-11T17:23:47Z |
Hi all,

I'm trying to import data from Twitter using the Gnip Power Track Twitter Import, and I'm getting the following error:

2013-01-11 08:57:51,846 INFO ActionStartXCommand:539 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack Start action 0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval 10
2013-01-11 08:57:56,990 WARN JavaActionExecutor:542 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack credentials is null for the action
2013-01-11 08:57:56,995 WARN JavaActionExecutor:542 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack Could not find credentials properties for: null
2013-01-11 08:58:06,172 INFO JavaActionExecutor:539 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack checking action, external ID job_201301110853_0001 status RUNNING
2013-01-11 08:58:06,216 WARN ActionStartXCommand:542 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack ***0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack***Action status=RUNNING
2013-01-11 08:58:06,510 WARN ActionStartXCommand:542 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack ***0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack***Action updated in DB!
2013-01-11 08:58:24,853 INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack callback for action 0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack
2013-01-11 08:58:26,165 INFO JavaActionExecutor:539 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack action completed, external ID job_201301110853_0001
2013-01-11 08:58:26,415 WARN JavaActionExecutor:542 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack Launcher ERROR, reason: Main class http://com.ibm.sda.ingest.SDAIngest, main() threw exception, Application submission failed.
2013-01-11 08:58:26,415 WARN JavaActionExecutor:542 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack Launcher exception: Application submission failed.
com.ibm.sda.common.SDAException: Application submission failed.
at com.ibm.sda.common.JobLife.runJob(JobLife.java:74)
at com.ibm.sda.ingest.SDAIngest.main(SDAIngest.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1144)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

2013-01-11 08:58:27,062 INFO ActionEndXCommand:539 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION0000000-130111085534137-oozie-biad-W@ingest_gnip_powertrack ERROR is considered as FAILED for SLA
2013-01-11 08:58:28,051 WARN CoordActionUpdateXCommand:542 - USERbiadmin GROUPsupergroup] TOKEN[ APP_SDA_Ingest_Gnip_Powertrack JOB0000000-130111085534137-oozie-biad-W ACTION[-] E1100: Command precondition does not hold before execution, , coord action is null, Error Code: E1100

You can see below how I have filled the properties and rules files:

Properties Files:
gnipupdateurl=https://api.gnip.com:443/accounts/XXXX/publishers/twitter/streams/track/Prod/rules.json
gnipuser=XXXX
gnipurl=https://stream.gnip.com:443/accounts/XXXX/publishers/twitter/streams/track/Prod.json
gnippassword=XXXX
gnipmaxreplayrange=

Rules Files: {"rules":}

Could anyone help?

Regards,
Tiago Moraes
Updated on 2013-03-21T12:28:50Z at 2013-03-21T12:28:50Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    603 Posts

    Re: Gnip PowerTrack Import Error

    ‏2013-01-14T17:51:54Z  
    Hi Tiago,

    1) Please make sure to tune up the memory params in the mapred-site.xml per instructions below:

    http://pic.dhe.ibm.com/infocenter/bigins/v2r0/topic/com.ibm.swg.im.infosphere.biginsights.install.doc/doc/acc_mda_configure.html

    2) Also try restarting the BigInsights cluster and try the SDA app again.

    Hope this helps.

    Thank you!

    Zach
  • SystemAdmin
    SystemAdmin
    603 Posts

    Re: Gnip PowerTrack Import Error

    ‏2013-03-21T12:28:50Z  
    Hi Tiago,

    1) Please make sure to tune up the memory params in the mapred-site.xml per instructions below:

    http://pic.dhe.ibm.com/infocenter/bigins/v2r0/topic/com.ibm.swg.im.infosphere.biginsights.install.doc/doc/acc_mda_configure.html

    2) Also try restarting the BigInsights cluster and try the SDA app again.

    Hope this helps.

    Thank you!

    Zach
    As mentioned I have tuned up memory params in mapred-site.xml. I have also restarted hadoop cluster.
    But i am still facing below application submission error-

    Please help me out with this. I am stuck long with this. Its urgent

    2013-03-21 08:22:34,728 INFO ActionStartXCommand:539 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 Start action 0000000-130321081951802-oozie-biad-W@action1 with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval 10
    2013-03-21 08:22:40,365 WARN JavaActionExecutor:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 credentials is null for the action
    2013-03-21 08:22:40,372 WARN JavaActionExecutor:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 Could not find credentials properties for: null
    2013-03-21 08:22:50,365 INFO JavaActionExecutor:539 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 checking action, external ID job_201303210817_0001 status RUNNING
    2013-03-21 08:22:50,411 WARN ActionStartXCommand:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 ***0000000-130321081951802-oozie-biad-W@action1***Action status=RUNNING
    2013-03-21 08:22:50,538 WARN ActionStartXCommand:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 ***0000000-130321081951802-oozie-biad-W@action1***Action updated in DB!
    2013-03-21 08:24:52,369 INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 callback for action 0000000-130321081951802-oozie-biad-W@action1
    2013-03-21 08:24:53,861 INFO JavaActionExecutor:539 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 action completed, external ID job_201303210817_0001
    2013-03-21 08:24:54,223 WARN JavaActionExecutor:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 Launcher ERROR, reason: Main class http://com.ibm.sda.ingest.SDAIngest, main() threw exception, Application submission failed.
    2013-03-21 08:24:54,223 WARN JavaActionExecutor:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 Launcher exception: Application submission failed.
    com.ibm.sda.common.SDAException: Application submission failed.
    at com.ibm.sda.common.JobLife.runJob(JobLife.java:74)
    at com.ibm.sda.ingest.SDAIngest.main(SDAIngest.java:607)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(AccessController.java:310)
    at javax.security.auth.Subject.doAs(Subject.java:573)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1144)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

    2013-03-21 08:24:55,243 INFO ActionEndXCommand:539 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 ERROR is considered as FAILED for SLA
    2013-03-21 08:24:56,938 WARN CoordActionUpdateXCommand:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION[-] E1100: Command precondition does not hold before execution, , coord action is null, Error Code: E1100
  • Frank_Blau
    Frank_Blau
    1 Post

    Re: Gnip PowerTrack Import Error

    ‏2013-12-06T00:06:05Z  
    As mentioned I have tuned up memory params in mapred-site.xml. I have also restarted hadoop cluster.
    But i am still facing below application submission error-

    Please help me out with this. I am stuck long with this. Its urgent

    2013-03-21 08:22:34,728 INFO ActionStartXCommand:539 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 Start action 0000000-130321081951802-oozie-biad-W@action1 with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval 10
    2013-03-21 08:22:40,365 WARN JavaActionExecutor:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 credentials is null for the action
    2013-03-21 08:22:40,372 WARN JavaActionExecutor:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 Could not find credentials properties for: null
    2013-03-21 08:22:50,365 INFO JavaActionExecutor:539 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 checking action, external ID job_201303210817_0001 status RUNNING
    2013-03-21 08:22:50,411 WARN ActionStartXCommand:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 ***0000000-130321081951802-oozie-biad-W@action1***Action status=RUNNING
    2013-03-21 08:22:50,538 WARN ActionStartXCommand:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 ***0000000-130321081951802-oozie-biad-W@action1***Action updated in DB!
    2013-03-21 08:24:52,369 INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 callback for action 0000000-130321081951802-oozie-biad-W@action1
    2013-03-21 08:24:53,861 INFO JavaActionExecutor:539 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 action completed, external ID job_201303210817_0001
    2013-03-21 08:24:54,223 WARN JavaActionExecutor:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 Launcher ERROR, reason: Main class http://com.ibm.sda.ingest.SDAIngest, main() threw exception, Application submission failed.
    2013-03-21 08:24:54,223 WARN JavaActionExecutor:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 Launcher exception: Application submission failed.
    com.ibm.sda.common.SDAException: Application submission failed.
    at com.ibm.sda.common.JobLife.runJob(JobLife.java:74)
    at com.ibm.sda.ingest.SDAIngest.main(SDAIngest.java:607)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(AccessController.java:310)
    at javax.security.auth.Subject.doAs(Subject.java:573)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1144)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

    2013-03-21 08:24:55,243 INFO ActionEndXCommand:539 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION0000000-130321081951802-oozie-biad-W@action1 ERROR is considered as FAILED for SLA
    2013-03-21 08:24:56,938 WARN CoordActionUpdateXCommand:542 - USERbiadmin GROUP-] TOKEN[ APP_BoardReader_Blogs_Ingest JOB0000000-130321081951802-oozie-biad-W ACTION[-] E1100: Command precondition does not hold before execution, , coord action is null, Error Code: E1100

    I am also getting this error and I have also tuned up the memory settings.

     

    Any help would be appreciated.

  • Debabrata.Ghosh
    Debabrata.Ghosh
    5 Posts

    Re: Gnip PowerTrack Import Error

    ‏2013-12-11T06:03:50Z  

    I am also getting this error and I have also tuned up the memory settings.

     

    Any help would be appreciated.

    SDA supports Gnip Twitter data in its original format . There is another option about fetching the data in Gnip-normalized stream format but SDA doesn't support that. So, please ensure the appropriate option is select in the Settings within Gnip account.

    Further in the rules file, you may need to set some handle based on which you would want to retrieve the twitter data.

    In case , if the above mentioned options still doesn't work, then please check the Task Log. It's highly likely that the configuration of Streams may be missing a step. If you execute the command "streamtool lsjobs -i <instance id>" and it asks for a password then you would need to execute the command "streamtool genkey" for password less access.