CentOS6.5實現Hadoop_2.8.1編譯
CentOS6.5實現Hadoop編譯
Hadoop 2.x 三大元件
MapReduce(Others) -- 計算
YARN -- 資源和作業排程平臺
HDFS -- 儲存
環境準備：
[root@hadoop001 software]# pwd
/opt/software
-rw-r--r--. 1 root root 8617253 May 14 07:05 apache-maven-3.3.9-bin.zip
-rw-r--r--. 1 root root 7546219 May 14 07:05 findbugs-1.3.9.zip
-rw-r--r--. 1 root root 34523353 May 14 07:05 hadoop-2.8.1-src.tar.gz
-rw-r--r--. 1 root root 424555111 May 14 07:09 hadoop-2.8.1.tar.gz
-rw-r--r--. 1 root root 173271626 May 14 07:09 jdk-8u45-linux-x64.gz
-rw-r--r--. 1 root root 96721446 May 14 07:07 .m2.tar.gz
-rw-r--r--. 1 root root 2401901 May 14 07:04 protobuf-2.5.0.tar.gz
1.Hadoop原始碼下載
[root@hadoop001 software]# tar -xzvf hadoop-2.8.1-src.tar.gz
# 檢查原始碼包環境
[root@hadoop001 hadoop-2.8.1-src]# cat BUILDING.txt
----------------------------------------------------------------------------------
Requirements:
* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
2.JAVA安裝
[root@hadoop001 ~]# mkdir -p /usr/java
[root@hadoop001 ~]# mv jdk-8u45-linux-x64.gz /usr/java
[root@hadoop001 ~]# cd /usr/java
[root@hadoop001 ~]# tar -xzvf jdk-8u45-linux-x64.gz
# 修改使用者和使用者組
[root@hadoop001 java]# ll
total 169388
drwxr-xr-x. 8 uucp 143 4096 Apr 10 2015 jdk1.8.0_45
-rw-r--r--. 1 root root 173271626 May 14 07:09 jdk-8u45-linux-x64.gz
[root@hadoop002 java]# chown -R root:root jdk1.8.0_45
[root@hadoop002 java]# ll
total 169388
drwxr-xr-x. 8 root root 4096 Apr 11 2015 jdk1.8.0_45
-rw-r--r--. 1 root root 173271626 Mar 16 15:25 jdk-8u45-linux-x64.gz
# JAVA全域性變數配置
[root@hadoop001 java]# vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
注：不用解除安裝原有JDK,環境變數可以覆蓋原來JDK
[root@hadoop001 java]# source /etc/profile
[root@hadoop001 java]# which java
/usr/java/jdk1.8.0_45/bin/java
[root@hadoop001 java]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
3.Maven安裝與部署
[root@hadoop001 software]# ll
total 466948
-rw-r--r--. 1 root root 8617253 May 14 07:05 apache-maven-3.3.9-bin.zip
[root@hadoop001 software]# unzip apache-maven-3.3.9-bin.zip
[root@hadoop001 software]# ll
total 466952
drwxr-xr-x. 6 root root 4096 Nov 10 2015 apache-maven-3.3.9
-rw-r--r--. 1 root root 8617253 May 14 07:05 apache-maven-3.3.9-bin.zip
#Maven全域性環境變數
[root@hadoop001 java]# vi /etc/profile
export MAVEN_HOME=/opt/software/apache-maven-3.3.9
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 software]# source /etc/profile
[root@hadoop001 software]# which mvn
/opt/software/apache-maven-3.3.9/bin/mvn
[root@hadoop001 software]# mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
#Maven倉庫檔案手動匯入
[root@hadoop001 software]# mv .m2.tar.gz ~
[root@hadoop001 software]# cd ~
[root@hadoop001 ~]# tar -xzvf .m2.tar.gz
# Maven倉庫預設路徑和修改
[root@hadoop001 conf]# pwd
/opt/software/apache-maven-3.3.9/conf/settings.xml
<!-- localRepository
| The path to the local repository maven will use to store artifacts.
|
| Default: ${user.home}/.m2/repository
/path/to/local/repo
-->
-- pro.xml以maven倉庫的專案，mvn 編譯、打包、測試
4.protobuf原始碼安裝
[root@hadoop001 software]# pwd
/opt/software
[root@hadoop001 software]# tar -xzvf protobuf-2.5.0.tar.gz
[root@hadoop001 software]# ll
total 466956
drwxr-xr-x. 10 109965 5000 4096 Feb 26 2013 protobuf-2.5.0
-rw-r--r--. 1 root root 2401901 May 14 07:04 protobuf-2.5.0.tar.gz
[root@hadoop001 software]# chown -R root:root protobuf-2.5.0
[root@hadoop001 software]# cd protobuf-2.5.0
[root@hadoop001 protobuf-2.5.0]# yum install -y gcc gcc-c++ make cmake
[root@hadoop001 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf
[root@hadoop001 protobuf-2.5.0]# make && make install
# protobuf全域性環境變數
[root@hadoop001 java]# vi /etc/profile
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 protobuf-2.5.0]# source /etc/profile
[root@hadoop001 protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
[root@hadoop001 protobuf-2.5.0]#
5.Findbugs安裝
[root@hadoop001 software]# pwd
/opt/software
[root@hadoop001 software]# unzip findbugs-1.3.9.zip
[root@hadoop001 software]# ll
total 466960
drwxr-xr-x. 7 root root 4096 Aug 21 2009 findbugs-1.3.9
-rw-r--r--. 1 root root 7546219 May 14 07:05 findbugs-1.3.9.zip
#Findbugs全域性環境變數
[root@hadoop002 software]# vi /etc/profile
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 software]#
[root@hadoop001 software]# source /etc/profile
[root@hadoop001 software]# findbugs -version
1.3.9
6.其他依賴
yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake
7.Hadoop原始碼編譯
[root@hadoop001 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 3.050 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 7.795 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 5.657 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 6.914 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.264 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 7.341 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.698 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 6.050 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 6.549 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 3.666 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.016 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 47.066 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:39 min
[INFO] Finished at: 2018-05-14T22:30:38-04:00
[INFO] Final Memory: 190M/454M
[INFO]
Hadoop 2.x 三大元件
MapReduce(Others) -- 計算
YARN -- 資源和作業排程平臺
HDFS -- 儲存
環境準備：
[root@hadoop001 software]# pwd
/opt/software
-rw-r--r--. 1 root root 8617253 May 14 07:05 apache-maven-3.3.9-bin.zip
-rw-r--r--. 1 root root 7546219 May 14 07:05 findbugs-1.3.9.zip
-rw-r--r--. 1 root root 34523353 May 14 07:05 hadoop-2.8.1-src.tar.gz
-rw-r--r--. 1 root root 424555111 May 14 07:09 hadoop-2.8.1.tar.gz
-rw-r--r--. 1 root root 173271626 May 14 07:09 jdk-8u45-linux-x64.gz
-rw-r--r--. 1 root root 96721446 May 14 07:07 .m2.tar.gz
-rw-r--r--. 1 root root 2401901 May 14 07:04 protobuf-2.5.0.tar.gz
1.Hadoop原始碼下載
[root@hadoop001 software]# tar -xzvf hadoop-2.8.1-src.tar.gz
# 檢查原始碼包環境
[root@hadoop001 hadoop-2.8.1-src]# cat BUILDING.txt
----------------------------------------------------------------------------------
Requirements:
* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
2.JAVA安裝
[root@hadoop001 ~]# mkdir -p /usr/java
[root@hadoop001 ~]# mv jdk-8u45-linux-x64.gz /usr/java
[root@hadoop001 ~]# cd /usr/java
[root@hadoop001 ~]# tar -xzvf jdk-8u45-linux-x64.gz
# 修改使用者和使用者組
[root@hadoop001 java]# ll
total 169388
drwxr-xr-x. 8 uucp 143 4096 Apr 10 2015 jdk1.8.0_45
-rw-r--r--. 1 root root 173271626 May 14 07:09 jdk-8u45-linux-x64.gz
[root@hadoop002 java]# chown -R root:root jdk1.8.0_45
[root@hadoop002 java]# ll
total 169388
drwxr-xr-x. 8 root root 4096 Apr 11 2015 jdk1.8.0_45
-rw-r--r--. 1 root root 173271626 Mar 16 15:25 jdk-8u45-linux-x64.gz
# JAVA全域性變數配置
[root@hadoop001 java]# vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
注：不用解除安裝原有JDK,環境變數可以覆蓋原來JDK
[root@hadoop001 java]# source /etc/profile
[root@hadoop001 java]# which java
/usr/java/jdk1.8.0_45/bin/java
[root@hadoop001 java]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
3.Maven安裝與部署
[root@hadoop001 software]# ll
total 466948
-rw-r--r--. 1 root root 8617253 May 14 07:05 apache-maven-3.3.9-bin.zip
[root@hadoop001 software]# unzip apache-maven-3.3.9-bin.zip
[root@hadoop001 software]# ll
total 466952
drwxr-xr-x. 6 root root 4096 Nov 10 2015 apache-maven-3.3.9
-rw-r--r--. 1 root root 8617253 May 14 07:05 apache-maven-3.3.9-bin.zip
#Maven全域性環境變數
[root@hadoop001 java]# vi /etc/profile
export MAVEN_HOME=/opt/software/apache-maven-3.3.9
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 software]# source /etc/profile
[root@hadoop001 software]# which mvn
/opt/software/apache-maven-3.3.9/bin/mvn
[root@hadoop001 software]# mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
#Maven倉庫檔案手動匯入
[root@hadoop001 software]# mv .m2.tar.gz ~
[root@hadoop001 software]# cd ~
[root@hadoop001 ~]# tar -xzvf .m2.tar.gz
# Maven倉庫預設路徑和修改
[root@hadoop001 conf]# pwd
/opt/software/apache-maven-3.3.9/conf/settings.xml
<!-- localRepository
| The path to the local repository maven will use to store artifacts.
|
| Default: ${user.home}/.m2/repository
/path/to/local/repo
-->
-- pro.xml以maven倉庫的專案，mvn 編譯、打包、測試
4.protobuf原始碼安裝
[root@hadoop001 software]# pwd
/opt/software
[root@hadoop001 software]# tar -xzvf protobuf-2.5.0.tar.gz
[root@hadoop001 software]# ll
total 466956
drwxr-xr-x. 10 109965 5000 4096 Feb 26 2013 protobuf-2.5.0
-rw-r--r--. 1 root root 2401901 May 14 07:04 protobuf-2.5.0.tar.gz
[root@hadoop001 software]# chown -R root:root protobuf-2.5.0
[root@hadoop001 software]# cd protobuf-2.5.0
[root@hadoop001 protobuf-2.5.0]# yum install -y gcc gcc-c++ make cmake
[root@hadoop001 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf
[root@hadoop001 protobuf-2.5.0]# make && make install
# protobuf全域性環境變數
[root@hadoop001 java]# vi /etc/profile
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 protobuf-2.5.0]# source /etc/profile
[root@hadoop001 protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
[root@hadoop001 protobuf-2.5.0]#
5.Findbugs安裝
[root@hadoop001 software]# pwd
/opt/software
[root@hadoop001 software]# unzip findbugs-1.3.9.zip
[root@hadoop001 software]# ll
total 466960
drwxr-xr-x. 7 root root 4096 Aug 21 2009 findbugs-1.3.9
-rw-r--r--. 1 root root 7546219 May 14 07:05 findbugs-1.3.9.zip
#Findbugs全域性環境變數
[root@hadoop002 software]# vi /etc/profile
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop001 software]#
[root@hadoop001 software]# source /etc/profile
[root@hadoop001 software]# findbugs -version
1.3.9
6.其他依賴
yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake
7.Hadoop原始碼編譯
[root@hadoop001 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 3.050 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 7.795 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 5.657 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 6.914 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.264 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 7.341 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.698 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 6.050 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 6.549 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 3.666 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.016 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 47.066 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:39 min
[INFO] Finished at: 2018-05-14T22:30:38-04:00
[INFO] Final Memory: 190M/454M
[INFO]
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/31441024/viewspace-2154525/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- CentOS6.5實現Hadoop_2.8.1編譯及HDFS偽分散式部署CentOSHadoop編譯分散式
- Mysql5.6 for Centos6.5原始碼編譯安裝MySqlCentOS原始碼編譯
- 從零實現最簡編譯模型編譯模型
- Elam的caffe筆記之配置篇（一）：CentOS6.5編譯安裝gcc4.8.2筆記CentOS編譯GC
- 實現一個簡單的 JavaScript 編譯器JavaScript編譯
- UE4 Shader 編譯以及變種實現編譯
- 基於.net standard 的動態編譯實現編譯
- 實現交叉編譯gdb和gdbserver(gdb-7.6)編譯Server
- Java 實現《編譯原理》簡單詞法分析功能Java編譯原理詞法分析
- 手把手教你實現Android編譯期註解Android編譯
- 編譯提速最佳實踐編譯
- 使用gulp和bable實現實時編譯ES6程式碼編譯
- CentOS 8上安裝GCC實現開發編譯功能CentOSGC編譯
- Ubuntu20.04linux核心(5.4.0版本)編譯準備與實現過程-編譯過程（2）UbuntuLinux編譯
- ☕【Java技術指南】「編譯器專題」深入分析探究“靜態編譯器”（JAVA\IDEA\ECJ編譯器）是否可以實現程式碼優化？Java編譯Idea優化
- Vue（27）vue-codemirror實現線上程式碼編譯器Vue編譯
- Roslyn 編譯器Api妙用：動態生成類並實現介面ROS編譯API
- nginx 編譯出現的問題Nginx編譯
- Java編譯與反編譯Java編譯
- 編譯器實現之旅——第五章 實現語法分析器前的準備編譯語法分析
- Gitee Webhook 實現自動拉取程式碼並編譯程式碼GiteeWebHook編譯
- 將HTML字串編譯為虛擬DOM物件的基礎實現HTML字串編譯物件
- Taro編譯打包優化實踐編譯優化
- Atlas 2.1.0 實踐（1）—— 編譯Atlas編譯
- libusb android ndk編譯--編譯mipsAndroid編譯
- hadoop編譯—+2.x編譯Hadoop編譯
- N1064編譯鏈編譯編譯
- 編譯編譯
- vue3 模板編譯 —— 30 行程式碼實現 .sync 修飾符Vue編譯行程
- 實現微信小程式編譯和執行環境系列(初始篇)微信小程式編譯
- 透過 GitHub Actions 實現程式碼的自動編譯和釋出Github編譯
- 編譯實踐學習 Part1編譯
- 編譯實踐學習 Part2編譯
- 編譯實踐學習 Part5編譯
- CentOS6.5源CentOS
- 現代編譯原理C語言描述pdf編譯原理C語言
- 用javac編譯servlet類出現問題Java編譯Servlet
- 程式碼線上編譯器（上）- 編輯及編譯編譯