Sqoop fetchsize失效
前幾天線上Sqoop的一個Job(從MySQL抽取資料到Hadoop)突然報了OOME,後重跑並做java trace發現記憶體佔用都是byte[],同時cpu top 3的方法都是com.mysql.jdbc.ByteArraryBuffer.getBytes即記憶體都是被資料消耗掉了;很奇怪,為什麼在option裡指定了fetch-size=100怎麼會OOME呢(平均記錄長度不到1kb);
再看昨天成功的發現100W條記錄,發現佔用了860MB記憶體,明顯是fetch-size沒有生效
+---------+---------+------------+----------+-------------+--------------+
| type | status | host | cpusec | mrinput_rec | memory_mb |
+---------+---------+------------+----------+-------------+--------------+
| CLEANUP | SUCCESS | A | 0.3400 | NULL | 191.84765625 |
| MAP | SUCCESS | A | 335.6400 | 1006942 | 862.39843750 |
| SETUP | SUCCESS | B | 0.2000 | NULL | 179.34765625 |
+---------+---------+------------+----------+-------------+--------------+
沒辦法,把sqoop原始碼翻出來終於發現RC了:fetchsize被忽略掉了
protected void initOptionDefaults() {
if (options.getFetchSize() == null) {
LOG.info("Preparing to use a MySQL streaming resultset.");
options.setFetchSize(Integer.MIN_VALUE);
} else if (
!options.getFetchSize().equals(Integer.MIN_VALUE)
&& !options.getFetchSize().equals(0)) {
LOG.info("Argument '--fetch-size " + options.getFetchSize()
+ "' will probably get ignored by MySQL JDBC driver.");
}
}
究其原因是MySQL提供的API只支援row-by-row和all模式:
By default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate, and due to the design of the MySQL network protocol is easier to implement. If you are working with ResultSets that have a large number of rows or large values, and cannot allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time.
http://dev.mysql.com/doc/refman/5.5/en/connector-j-reference-implementation-notes.html
最後把fetchsize給去掉了,Job執行成功,700W行佔用記憶體400MB;
+---------+---------+------------+----------+-------------+--------------+
| type | status | host | cpusec | mrinput_rec | memory_mb |
+---------+---------+------------+----------+-------------+--------------+
| CLEANUP | SUCCESS | A | 0.4200 | NULL | 183.49218750 |
| MAP | FAILED | B | NULL | NULL | NULL |
| MAP | SUCCESS | A | 377.1200 | 7195560 | 408.08593750 |
| SETUP | SUCCESS | C | 0.2900 | NULL | 188.64843750 |
+---------+---------+------------+----------+-------------+--------------+
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/11676357/viewspace-1060920/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- Sqoop之 Sqoop 1.4.6 安裝OOP
- sqoopOOP
- Sqoop jobOOP
- Sqoop(1)OOP
- 【Sqoop篇】----Sqoop從搭建到應用案例OOP
- [20180511]PLSQL與fetchsize.txtSQL
- Sqoop同步策略OOP
- sqoop 的使用OOP
- SQOOP安裝部署OOP
- sqoop 架構分析OOP架構
- sqoop小作業OOP
- 【大資料】— sqoop ?大資料OOP
- sqoop的安裝OOP
- sqoop簡單使用OOP
- 4- sqoop語法OOP
- sqoop指令碼批量生成OOP指令碼
- 【Hadoop】9、Sqoop元件Hadoop元件
- Sqoop2安裝OOP
- Hadoop Sqoop介紹Hadoop
- Sqoop 並行抽數OOP並行
- sqoop的外掛oraoopOOP
- sqoop 安裝與配置使用OOP
- Sqoop1.4.6配置和使用OOP
- Sqoop1 From PostgreSQL to HdfsOOPSQL
- Sqoop安裝及驗證OOP
- sqoop應用例項1OOP
- openview失效View
- sqoop匯入orcle注意事項OOP
- sqoop如何指定pg庫的模式OOP模式
- Apache Sqoop與Apache Flume比較ApacheOOP
- 資料同步工具Sqoop和DataXOOP
- sqoop資料匯入匯出OOP
- Sqoop的介紹和安裝OOP
- Sqoop匯出ClickHouse資料到HiveOOPHive
- sqoop常用命令整理(一)OOP
- sqoop條件抽取報錯distcpOOPTCP
- Sqoop 1.99.6 安裝和使用OOP
- sqoop export 資料覆蓋更新OOPExport