Topic
No replies
Y7YG_春艳_贺
Y7YG_春艳_贺
3 Posts
ACCEPTED ANSWER

Pinned topic the problem encountered while using oozie to run sqoop job

‏2013-07-27T10:26:23Z |

I create a sqoop job to import data from oracle incremental by id,the command is :

sqoop job --create stujob -- import --hive-import --connect jdbc:oracle:thin:@192.168.1.8:1521:ORCL --username SCOTT --password tiger  -m 1 --table  STUDENT --check-column SID --incremental append --last-value 0

then I can run bin/sqoop job --exec stujob successfully,but the task can't run successfully in oozie

 

Controller.doPrivileged(AccessController.java:310)
        at javax.security.auth.Subject.doAs(Subject.java:573)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
2098 [main] ERROR org.apache.sqoop.manager.OracleManager  - Failed to list columns
java.sql.SQLException: ORA-01005: null password given; logon denied

        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:391)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:383)
        at oracle.jdbc.driver.T4CTTIfun.processError(T4CTTIfun.java:638)
        at oracle.jdbc.driver.T4CTTIoauthenticate.processError(T4CTTIoauthenticate.java:433)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:459)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:193)
        at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:365)
        at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:804)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:386)
        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:535)
        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:218)
        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:29)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:528)
        at java.sql.DriverManager.getConnection(DriverManager.java:402)
        at java.sql.DriverManager.getConnection(DriverManager.java:450)
        at org.apache.sqoop.manager.OracleManager.makeConnection(OracleManager.java:310)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.manager.OracleManager.getColumnNames(OracleManager.java:751)
        at org.apache.sqoop.orm.ClassWriter.getColumnNames(ClassWriter.java:1207)
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1062)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
        at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
        at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
        at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
        at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(AccessController.java:310)
        at javax.security.auth.Subject.doAs(Subject.java:573)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
2100 [main] ERROR org.apache.sqoop.tool.ImportTool  - Imported Failed: Attempted to generate class with no columns!
Intercepting System.exit(1)

It seems that the oozie can't know the password of the oracle define in the job,the document said :you have to set sqoop.metastore.client.record.password to true if you are executing saved jobs via Oozie because Sqoop cannot prompt the user to enter passwords while being executed as Oozie tasks.

I do have set the sqoop.metastore.client.record.password to be true

the sqoop-site.xml :

<configuration>

 
  <!--
    Set the value of this property to explicitly enable third-party
    ManagerFactory plugins.
 
    If this is not used, you can alternately specify a set of ManagerFactories
    in the $SQOOP_CONF_DIR/managers.d/ subdirectory.  Each file should contain
    one or more lines like:
      manager.class.name[=/path/to/containing.jar]
 
    Files will be consulted in lexicographical order only if this property
    is unset.
  -->
  <!--
  <property>
    <name>sqoop.connection.factories</name>
    <value>com.cloudera.sqoop.manager.DefaultManagerFactory</value>
    <description>A comma-delimited list of ManagerFactory implementations
      which are consulted, in order, to instantiate ConnManager instances
      used to drive connections to databases.
    </description>
  </property>
  -->
 
  <!--
    Set the value of this property to enable third-party tools.
 
    If this is not used, you can alternately specify a set of ToolPlugins
    in the $SQOOP_CONF_DIR/tools.d/ subdirectory.  Each file should contain
    one or more lines like:
      plugin.class.name[=/path/to/containing.jar]
 
    Files will be consulted in lexicographical order only if this property
    is unset.
  -->
  <!--
  <property>
    <name>sqoop.tool.plugins</name>
    <value></value>
    <description>A comma-delimited list of ToolPlugin implementations
      which are consulted, in order, to register SqoopTool instances which
      allow third-party tools to be used.
    </description>
  </property>
  -->
 
  <!--
    By default, the Sqoop metastore will auto-connect to a local embedded
    database stored in ~/.sqoop/. To disable metastore auto-connect, uncomment
    this next property.
  -->
 
  <property>
    <name>sqoop.metastore.client.enable.autoconnect</name>
    <value>true</value>
    <description>If true, Sqoop will connect to a local metastore
      for job management when no other metastore arguments are
      provided.
    </description>
  </property>
 
 
  <!--
    The auto-connect metastore is stored in ~/.sqoop/. Uncomment
    these next arguments to control the auto-connect process with
    greater precision.
  -->
  <!--
  <property>
    <name>sqoop.metastore.client.autoconnect.url</name>
    <value>jdbc:hsqldb:file:/tmp/sqoop-meta/meta.db;shutdown=true</value>
    <description>The connect string to use when connecting to a
      job-management metastore. If unspecified, uses ~/.sqoop/.
      You can specify a different path here.
    </description>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.username</name>
    <value>SA</value>
    <description>The username to bind to the metastore.
    </description>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.password</name>
    <value></value>
    <description>The password to bind to the metastore.
    </description>
  </property>
   -->
 
  <!--
    For security reasons, by default your database password will not be stored in
    the Sqoop metastore. When executing a saved job, you will need to
    reenter the database password. Uncomment this setting to enable saved
    password storage. (INSECURE!)
  -->
  
  <property>
    <name>sqoop.metastore.client.record.password</name>
    <value>true</value>
    <description>If true, allow saved passwords in the metastore.
    </description>
  </property>
 
 
 
  <!--
    SERVER CONFIGURATION: If you plan to run a Sqoop metastore on this machine,
    you should uncomment and set these parameters appropriately.
 
    You should then configure clients with:
       sqoop.metastore.client.autoconnect.url =
       jdbc:hsqldb:hsql://&lt;server-name&gt;:&lt;port&gt;/sqoop
  -->
 <!--
  <property>
    <name>sqoop.metastore.server.location</name>
    <value>/tmp/sqoop-metastore/shared.db</value>
    <description>Path to the shared metastore database files.
    If this is not set, it will be placed in ~/.sqoop/.
    </description>
  </property>
  -->
  <property>
    <name>sqoop.metastore.server.port</name>
    <value>16000</value>
    <description>Port that this metastore should listen on.
    </description>
  </property>
 
</configuration>
and the oozie workflow.xml is :

 

<workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf">
    <start to="sqoop-node"/>
 
    <action name="sqoop-node">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/biginsights/hive/warehouse/student"/>
            </prepare>
            <configuration>
             <property>
                    <name>sqoop.metastore.client.record.password</name>
                    <value>true</value>
                </property>
                <property>
                    <name>oozie.hive.defaults</name>
                    <value>my-hive-default.xml</value>
                </property>
                <property>
                    <name>hive.metastore.local</name>
                    <value>true</value>
                </property>
                <property>
                    <name>javax.jdo.option.ConnectionURL</name>
                    <value>jdbc:derby://BigInsightMaster:1528/commonDb;create=true</value>
                </property>
                <property>
                    <name>javax.jdo.option.ConnectionDriverName</name>
                    <value>org.apache.derby.jdbc.ClientDriver</value>
                </property>
                <property>
                    <name>javax.jdo.option.ConnectionUserName</name>
                    <value>HIVE</value>
                </property>
                <property>
                    <name>javax.jdo.option.ConnectionPassword</name>
                    <value>HIVE</value>
                </property>
                <property>
                    <name>hive.metastore.warehouse.dir</name>
                    <value>/biginsights/hive/warehouse</value>
                </property>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <arg>job</arg>
            <arg>--exec</arg>
            <arg>stujob</arg>
        </sqoop>
        <ok to="end"/>
        <error to="fail"/>
    </action>
 
    <kill name="fail">
        <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>
 

What shall I do to solve this problem?