sqoop job操作及執行Job時如何省略輸入MySQL儲存密碼

loveheping發表於2018-01-15
一、刪除Job

點選(此處)摺疊或開啟

  1. sqoop job \
  2. --delete testjob
二、建立Job
說明:在建立Job時,我們使用--password-file引數,而且非--passowrd。主要原因是在執行Job時使用--password引數將有警告,並且需要輸入密碼才能執行Job。當我們採用--password-file引數時,執行Job無需輸入資料庫密碼。

點選(此處)摺疊或開啟

  1. sqoop job \
  2. --create testjob \
  3. -- \
  4. import \
  5. --connect jdbc:mysql://192.168.137.130:3306/jepsondb \
  6. --username root \
  7. --password-file /input/sqoop.pwd \
  8. --table abc \
  9. --target-dir /input/abc \
  10. --delete-target-dir \
  11. -m 1
注:import與"--"之間必須要有一個空格

三、執行Job

點選(此處)摺疊或開啟

  1. sqoop job \
  2. --exec testjob
四、建立--password-file引數需要的密碼檔案(注:sqoop規定密碼檔案必須放在HDFS之上,並且許可權必須為400)
1、操作命令:

點選(此處)摺疊或開啟

  1. echo -n "123456" > sqoop.pwd
  2. hdfs dfs -rm sqoop.pwd /input/sqoop.pwd
  3. hdfs dfs -put sqoop.pwd /input
  4. hdfs dfs -chmod 400 /input/sqoop.pwd
  5. hdfs dfs -ls /input
  6. -r-------- 1 hadoop supergroup 6 2018-01-15 18:38 /input/sqoop.pwd
2、檢查sqoop的配置檔案(sqoop-site.xml)是否存在以下配置,並且value值為true

點選(此處)摺疊或開啟

  1. <property>
  2.     <name>sqoop.metastore.client.record.password</name>
  3.     <value>true</value>
  4.     <description>If true, allow saved passwords in the metastore.
  5.     </description>
  6. </property>
注:password檔案的格式不對錯誤提示如下:

點選(此處)摺疊或開啟

  1. arning: /app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.
  2. Please set $HBASE_HOME to the root of your HBase installation.
  3. Warning: /app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.
  4. Please set $HCAT_HOME to the root of your HCatalog installation.
  5. Warning: /app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.
  6. Please set $ACCUMULO_HOME to the root of your Accumulo installation.
  7. Warning: /app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.
  8. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
  9. 18/01/15 18:33:50 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
  10. 18/01/15 18:33:53 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
  11. 18/01/15 18:33:53 INFO tool.CodeGenTool: Beginning code generation
  12. 18/01/15 18:33:54 ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: Access denied for user 'root'@'spark220' (using password: YES)
  13. java.sql.SQLException: Access denied for user 'root'@'spark220' (using password: YES)
  14. at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
  15. at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
  16. at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
  17. at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)
  18. at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)
  19. at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
  20. at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2188)
  21. at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2219)
  22. at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2014)
  23. at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
  24. at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
  25. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  26. at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  27. at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  28. at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  29. at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
  30. at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
  31. at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
  32. at java.sql.DriverManager.getConnection(DriverManager.java:664)
  33. at java.sql.DriverManager.getConnection(DriverManager.java:247)
  34. at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:904)
  35. at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
  36. at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
  37. at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
  38. at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
  39. at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
  40. at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
  41. at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
  42. at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1846)
  43. at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1646)
  44. at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
  45. at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
  46. at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
  47. at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
  48. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  49. at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
  50. at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
  51. at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
  52. at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
  53. 18/01/15 18:33:54 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
  54. at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1652)
  55. at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
  56. at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
  57. at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
  58. at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
  59. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  60. at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
  61. at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
  62. at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
  63. at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
解決辦法參見第四點。

若澤大資料交流群:671914634

來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/31511218/viewspace-2150175/,如需轉載,請註明出處,否則將追究法律責任。

相關文章