sqoop條件抽取報錯distcp
Sqoop抽取語句如下:
/usr/local/sqoop-1.4.6/bin/sqoop import \
--connect jdbc:mysql://10.13.50.125:3306/longer \
--username sqoop \
--password XXXXXX \
--query " select * from tabname where scheduled_deptime between UNIX_TIMESTAMP('$v_begin_date') and UNIX_TIMESTAMP('$v_end_date') and \$CONDITIONS " \
--hive-database testdb\
--hive-table tabname _test \
--hive-import \
--split-by dynamic_id \
--target-dir /user/hive/sqptarget \
--bindir /home/hadoop/new_binddir \
--outdir /home/hadoop/new_binddir \
--direct \
-m 8
--------------------------------------------------------------------------------
報錯如下(片段):
18/07/06 15:25:41 INFO common.FileUtils: Source is 76566465 bytes. (MAX: 33554432)
18/07/06 15:25:41 INFO common.FileUtils: Launch distributed copy (distcp) job.
18/07/06 15:25:41 ERROR metadata.Hive: Failed to move: java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
Failed with exception java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
18/07/06 15:25:41 ERROR exec.Task: Failed with exception java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2895)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:3205)
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1920)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:364)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2084)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1755)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1463)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1181)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
--------------------------------------------------------------------------
原因: hive 最後移動資料的時候,需要呼叫hadoop-distcp-X.X.X.jar,
解決方法:只需要把$HADOOP_HOME/share/hadoop/tools/lib/hadoop-distcp-x.x.x.jar 複製 $HIVE_HOME/lib下面,重啟hive即可
/usr/local/sqoop-1.4.6/bin/sqoop import \
--connect jdbc:mysql://10.13.50.125:3306/longer \
--username sqoop \
--password XXXXXX \
--query " select * from tabname where scheduled_deptime between UNIX_TIMESTAMP('$v_begin_date') and UNIX_TIMESTAMP('$v_end_date') and \$CONDITIONS " \
--hive-database testdb\
--hive-table tabname _test \
--hive-import \
--split-by dynamic_id \
--target-dir /user/hive/sqptarget \
--bindir /home/hadoop/new_binddir \
--outdir /home/hadoop/new_binddir \
--direct \
-m 8
--------------------------------------------------------------------------------
報錯如下(片段):
18/07/06 15:25:41 INFO common.FileUtils: Source is 76566465 bytes. (MAX: 33554432)
18/07/06 15:25:41 INFO common.FileUtils: Launch distributed copy (distcp) job.
18/07/06 15:25:41 ERROR metadata.Hive: Failed to move: java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
Failed with exception java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
18/07/06 15:25:41 ERROR exec.Task: Failed with exception java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2895)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:3205)
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1920)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:364)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2084)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1755)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1463)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1181)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
--------------------------------------------------------------------------
原因: hive 最後移動資料的時候,需要呼叫hadoop-distcp-X.X.X.jar,
解決方法:只需要把$HADOOP_HOME/share/hadoop/tools/lib/hadoop-distcp-x.x.x.jar 複製 $HIVE_HOME/lib下面,重啟hive即可
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/9606353/viewspace-2157457/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- ef8 Contains 查詢條件 報錯 $ 附近錯誤AI
- Oracle 條件索引 case when 報錯解決方案Oracle索引
- 除錯——條件斷點除錯斷點
- EMNLP 2019 | 面向科學文獻的事實與條件聯合抽取
- Laravel 傳送郵件報錯Laravel
- 條件渲染
- Sqoop從Oracle抽數錯:IO Error: Connection resetOOPOracleError
- 報表查詢條件的 N 種使用方式
- Laravel5.8 傳送郵件報錯Laravel
- Excel 條件格式Excel
- openGauss 前提條件
- PL/SQL 條件SQL
- 屈服條件8
- Selenium等待條件
- 條件函式函式
- 條件語句
- react 條件渲染React
- If條件倒裝
- OGG 抽取程序註冊時報OGG-08221 ORA-00001錯誤總結
- excel條件格式怎麼設定 excel條件格式在哪裡Excel
- Sqoop jobOOP
- Hadoop源生實用工具之distcpHadoopTCP
- Condition條件變數變數
- javaScript條件語句JavaScript
- jquery 多條件匹配jQuery
- java-queryWrapper條件JavaAPP
- mysql條件查詢MySql
- DMSQL LIMIT限定條件SQLMIT
- Rust 條件編譯Rust編譯
- github條件搜尋Github
- impala 條件函式函式
- JavaScript 條件運算子JavaScript
- MongoDB查詢條件MongoDB
- springboot 條件裝配Spring Boot
- GO 條件語句Go
- switch拼接where條件
- pthread 條件變數thread變數
- doxygen 宏定義/宏編譯/條件編譯/預處理/預編譯 不處理、忽略條件、分析所有條件、滿足所有條件的方法編譯