Topic
  • 1 reply
  • Latest Post - ‏2013-07-23T01:56:10Z by Y7YG_春艳_贺
Y7YG_春艳_贺
Y7YG_春艳_贺
3 Posts

Pinned topic the problem encountered while using oozie to run sqoop

‏2013-07-17T03:02:52Z |

I am trying to use oozie to run sqoop job.First I want to import a table from oracle to hive and it succeed with out any problem,but when the sqoop command 

trigger by oozie,I get the error:

51370 [Thread-67] INFO  org.apache.sqoop.hive.HiveImport  - FAILED: Error in semantic analysis: Line 2:17 Path is not legal ''hdfs://biginsightmaster:9000/user/biadmin/STUDENT'': Move from: hdfs://biginsightmaster:9000/user/biadmin/STUDENT to: hdfs://BigInsightMaster:9000/biginsights/hive/warehouse/student is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
51404 [main] ERROR org.apache.sqoop.tool.ImportTool  - Encountered IOException running import job: java.io.IOException: Hive exited with status 10
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:364)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:314)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:226)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)

at org.apache.hadoop.mapred.Child.main(Child.java:249) 

The scheme is the hostname,The hostname of the namenode is BigInsightMaster .And the oozie job.properties' content is 

 

nameNode=hdfs://BigInsightMaster:9000
jobTracker=BigInsightMaster:9001
queueName=default
examplesRoot=examples
 
oozie.use.system.libpath=true
 
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/sqoop

I don't know why in this oozie job ,the scheme in oozie.action.dir.path  ,mapred.working.dir,mapred.job.tracker turned to be biginsightmaster

 I found that the hive will check the scheme and if the schemes are different,the error will throw out.

the check source code is 

 

// only in 'local' mode do we copy stuff from one place to another.
    // reject different scheme/authority in other cases.
    if (!isLocal
        && (!StringUtils.equals(fromURI.getScheme(), toURI.getScheme()) || !StringUtils
        .equals(fromURI.getAuthority(), toURI.getAuthority()))) {
      String reason = "Move from: " + fromURI.toString() + " to: "
          + toURI.toString() + " is not valid. "
          + "Please check that values for params \"default.fs.name\" and "
          + "\"hive.metastore.warehouse.dir\" do not conflict.";
      throw new SemanticException(ErrorMsg.ILLEGAL_PATH.getMsg(ast, reason));
}

Are there any  suggestions to solve this problem? 

Updated on 2013-07-17T06:08:52Z at 2013-07-17T06:08:52Z by Y7YG_春艳_贺
  • Y7YG_春艳_贺
    Y7YG_春艳_贺
    3 Posts

    Re: the problem encountered while using oozie to run sqoop

    ‏2013-07-23T01:56:10Z  

    I solved this problem :

     

    update derby
    ./ij
    connect 'jdbc:derby://BigInsightMaster:1528/commonDb;user=HIVE;password=HIVE';
     
    update DBS set DB_LOCATION_URI = replace(DB_LOCATION_URI, 'BigInsightMaster', 'biginsightmaster')
     
    update SDS set LOCATION =   replace(LOCATION, 'BigInsightMaster', 'biginsightmaster') where LOCATION like '%BigInsightMaster%';
    or update table student's location
    describe extended student  //check location and update location
    hive> alter table student set location 'hdfs://biginsightmaster:9000/biginsights/hive/warehouse/student';