hadoop本地庫與系統版本不一致引起的錯誤解決方法

參考了下列文章的方法；
本次採用的是hadoop 2.6.0，系統是oracle linux 6.6 x64,
發現hadoop要求glibc 2.14以上版本，oracle linux 6.6的glibc版本為2.12；oracle linux 7上為2.17
安裝過程出現報錯，錯誤內容如下：

2.Start NameNode daemon and DataNode daemon:
  $ sbin/start-dfs.sh
[root@oel6x64 hadoop]# sbin/start-dfs.sh
15/04/17 22:05:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/hadoop/logs/hadoop-root-namenode-oel6x64.out
localhost: starting datanode, logging to /usr/hadoop/logs/hadoop-root-datanode-oel6x64.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is b2:cb:4b:3a:73:ac:16:61:c7:27:fe:95:b6:57:2f:1e.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop/logs/hadoop-root-secondarynamenode-oel6x64.out
15/04/17 22:07:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable





轉：http://blog.csdn.net/jiedushi/article/details/7496327

部署hadoop的叢集環境為  

作業系統 centos 5.8    

hadoop版本為cloudera   hadoop-0.20.2-cdh3u3

叢集中設定支援gzip lzo壓縮後，在對壓縮檔案進行讀取或者對輸入檔案壓縮的時候要使用到hadoop的本地庫，本地庫的預設位置在

$HADOOP_HOME/lib/native/Linux-amd64-64   (64位作業系統)

$HADOOP_HOME/lib/native/Linux-i386-32   （32位作業系統）

資料夾中的libhadoop.so檔案，就是hadoop的本地庫。

如果本地庫不存在，或者本地庫與當前作業系統的版本不一致的時候，會報下面的錯誤：

11/09/20 17:29:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

增加除錯資訊設定

$ export HADOOP_ROOT_LOGGER=DEBUG,console

$ hadoop fs -text /test/data/origz/access.log.gz

2012-04-24 15:55:43,269 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

error libhadoop.so  /lib64/libc.so.6 required (libc 2.6)    /usr/local/hadoop/lib/native/Linux-amd64-64  

說明系統中的glibc的版本和libhadoop.so需要的版本不一致導致

檢視系統的libc版本  

# ll /lib64/libc.so.6

lrwxrwxrwx 1 root root 11 Apr 24 16:49 /lib64/libc.so.6 -> libc-2.5.so

系統中的版本為2.5

將系統中的glibc升級為2.9

下載glibc
wget 

下載glibc-linuxthreads
wget

解壓
$tar -jxvf glibc-2.9.tar.bz2
$cd glibc-2.9
$tar -jxvf ../glibc-linuxthreads-2.5.tar.bz2
$cd ..
$export CFLAGS="-g -O2"
$./glibc-2.9/configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
$make
#make install

安裝編譯過程中需要注意三點：
1、要將glibc-linuxthreads解壓到glibc目錄下。
2、不能在glibc當前目錄下執行configure。
3、加上最佳化開關，export CFLAGS="-g -O2"，否則會出現錯誤


安裝完後，可以檢視ls -l /lib/libc.so.6已升級

lrwxrwxrwx 1 root root 11 Apr 24 16:49 /lib64/libc.so.6 -> libc-2.9.so

測試本地庫是否升級

$ export HADOOP_ROOT_LOGGER=DEBUG,console

$ hadoop fs -text /test/data/origz/access.log.gz

12/04/25 08:54:47 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
12/04/25 08:54:47 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
12/04/25 08:54:47 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/04/25 08:54:47 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
12/04/25 08:54:47 DEBUG fs.FSInputChecker: DFSClient readChunk got seqno 0 offsetInBlock 0 lastPacketInBlock false packetLen 132100
可以看到將glibc升級後不再報錯，已經成功載入本地庫