sqoop job操作及執行Job時如何省略輸入MySQL儲存密碼
一、刪除Job
二、建立Job
說明:在建立Job時,我們使用--password-file引數,而且非--passowrd。主要原因是在執行Job時使用--password引數將有警告,並且需要輸入密碼才能執行Job。當我們採用--password-file引數時,執行Job無需輸入資料庫密碼。
注:import與"--"之間必須要有一個空格
三、執行Job
四、建立--password-file引數需要的密碼檔案(注:sqoop規定密碼檔案必須放在HDFS之上,並且許可權必須為400)
1、操作命令:
2、檢查sqoop的配置檔案(sqoop-site.xml)是否存在以下配置,並且value值為true
注:password檔案的格式不對錯誤提示如下:
解決辦法參見第四點。
若澤大資料交流群:671914634
點選(此處)摺疊或開啟
- sqoop job \
- --delete testjob
說明:在建立Job時,我們使用--password-file引數,而且非--passowrd。主要原因是在執行Job時使用--password引數將有警告,並且需要輸入密碼才能執行Job。當我們採用--password-file引數時,執行Job無需輸入資料庫密碼。
點選(此處)摺疊或開啟
- sqoop job \
- --create testjob \
- -- \
- import \
- --connect jdbc:mysql://192.168.137.130:3306/jepsondb \
- --username root \
- --password-file /input/sqoop.pwd \
- --table abc \
- --target-dir /input/abc \
- --delete-target-dir \
- -m 1
三、執行Job
點選(此處)摺疊或開啟
- sqoop job \
- --exec testjob
1、操作命令:
點選(此處)摺疊或開啟
- echo -n "123456" > sqoop.pwd
- hdfs dfs -rm sqoop.pwd /input/sqoop.pwd
- hdfs dfs -put sqoop.pwd /input
- hdfs dfs -chmod 400 /input/sqoop.pwd
- hdfs dfs -ls /input
- -r-------- 1 hadoop supergroup 6 2018-01-15 18:38 /input/sqoop.pwd
點選(此處)摺疊或開啟
-
<property>
-
<name>sqoop.metastore.client.record.password</name>
-
<value>true</value>
-
<description>If true, allow saved passwords in the metastore.
-
</description>
- </property>
點選(此處)摺疊或開啟
- arning: /app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.
- Please set $HBASE_HOME to the root of your HBase installation.
- Warning: /app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.
- Please set $HCAT_HOME to the root of your HCatalog installation.
- Warning: /app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.
- Please set $ACCUMULO_HOME to the root of your Accumulo installation.
- Warning: /app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.
- Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
- 18/01/15 18:33:50 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
- 18/01/15 18:33:53 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
- 18/01/15 18:33:53 INFO tool.CodeGenTool: Beginning code generation
- 18/01/15 18:33:54 ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: Access denied for user 'root'@'spark220' (using password: YES)
- java.sql.SQLException: Access denied for user 'root'@'spark220' (using password: YES)
- at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
- at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
- at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
- at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)
- at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)
- at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
- at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2188)
- at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2219)
- at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2014)
- at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776)
- at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
- at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
- at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
- at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
- at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
- at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
- at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
- at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
- at java.sql.DriverManager.getConnection(DriverManager.java:664)
- at java.sql.DriverManager.getConnection(DriverManager.java:247)
- at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:904)
- at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
- at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
- at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
- at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
- at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
- at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
- at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
- at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1846)
- at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1646)
- at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
- at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
- at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
- at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
- at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
- at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
- at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
- at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
- at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
- 18/01/15 18:33:54 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
- at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1652)
- at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
- at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
- at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
- at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
- at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
- at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
- at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
- at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
- at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
若澤大資料交流群:671914634
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/31511218/viewspace-2150175/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- Oracle:建立JOB定時執行儲存過程Oracle儲存過程
- Sqoop jobOOP
- oracle建立job並執行jobOracle
- Oracle中如何停用JOB及如何使用JOBOracle
- oracle job使用詳解及job不執行的檢查方法Oracle
- Oracle及SQL Server Job執行時間討論及夏令時問題OracleSQLServer
- job任務均不執行,手工執行報job now running
- 採用job定時執行recover datafile
- Python程式碼解析: job = next(job for job in jobs if job.job_id == job_id)Python
- Job不能執行原因分析
- 停掉正在執行的JOB
- Elasticjob執行job冪等AST
- 建立job進行重新整理物化檢視,job執行失敗。
- ORACLE查詢JOB資訊及JOB建立Oracle
- Oracle Job 定時執行設定例項[final]Oracle
- Job長時間執行不能退出的分析
- 利用sqlserver Job 及內嵌儲存過程傳送郵件SQLServer儲存過程
- mysql 定時指令碼(event),類似oracle的jobMySql指令碼Oracle
- job 執行時間比排程間隔時間長
- Elastic-Job的執行原理及優化實踐AST優化
- job呼叫儲存過程的注意事項儲存過程
- 【RAC】rac中如何指定job的執行例項
- crontab 每隔 1 小時 2 小時的執行 job 寫法
- 時區調整對job的執行時間的影響
- 如何加密傳輸和儲存使用者密碼加密密碼
- spark job執行引數優化Spark優化
- 定義JOB的執行間隔
- saltstack非同步執行命令: job ID非同步
- 物化檢視job無法執行
- 如何在 Linux 中不輸入密碼執行 sudo 命令Linux密碼
- DBMS_JOB.SUBMIT 建立job定時排程MIT
- 【實戰】oracle job + 儲存過程 的使用示例Oracle儲存過程
- oracle中Job的操作Oracle
- 使ssh不用輸入密碼(指令碼執行scp)密碼指令碼
- Jerkins中job執行進行HTTP通知HTTP
- 使用Elastic Job的時候報“Job conflict with register center”,如何處理?AST
- Oracle中job無法自動執行Oracle
- Kill執行中的JOB的問題