hadoop64位系統編譯

燤禕發表於2020-12-06
Hadoop編譯

hadoop原始碼編譯64位系統版本

寫在前面：

因為Hadoop不提供64位編譯好的版本，只能用原始碼自行編譯64位版本，所以本文以hadoop2.6.0為例進行原始碼編譯。值得注意的是如果想編譯2.6.0版本的hadoop需要用JDK7，如果用JDK8編譯會報這樣的錯誤:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while creating archive:

[ERROR] Exit code: 1 - /home/hadoop/hadoop-2.5.2-src/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java:27: 錯誤: 意外的結束標記: </ul>
[ERROR] * </ul>
[ERROR] ^
[ERROR]
[ERROR] Command line was: /usr/java/jre/../bin/javadoc @options @packages
[ERROR]
[ERROR] Refer to the generated Javadoc files in '/home/hadoop/hadoop-2.5.2-src/hadoop-common-project/hadoop-annotations/target' dir.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-annotations

這是JDK版本引起的，而高版本的hadoop(如2.7.2是可以用JDK8成功編譯的)。不過編譯完成後的hadoop可以用JDK8執行。總的來說就是想編譯高版本的hadoop就用適當高版本的JDK。 還有一點是官網下載的tar.gz直接能用的安裝包是不能直接編譯的，需要對-src.tar.gz進行編譯。

具體步驟：

  1. 安裝JDK,配置環境變數
  2. 安裝maven，先解壓，再配置環境變數。網不好的話最好用阿里雲映象
cd /opt/install/apache-maven-3.0.5
vi conf/settings.xml

找到mirros標籤加入：
<mirrors>
    <!-- mirror
     | Specifies a repository mirror site to use instead of a given repository. The repository that
     | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
     | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
     |
<mirror>
       <id>mirrorId</id>
       <mirrorOf>repositoryId</mirrorOf>
       <name>Human Readable Name for this Mirror.</name>
       <url>http://my.repository.com/repo/path</url>
      </mirror>
     -->
        <mirror>
                <id>nexus-aliyun</id>
                <mirrorOf>central</mirrorOf>
                <name>Nexus aliyun</name>
                <url>http://maven.aliyun.com/nexus/content/groups/public</url>
        </mirror>
</mirrors>

最後不要忘記配置maven環境變數

vi /etc/profile

#MAVEN_HOME
export MAVEN_HOME=/opt/install/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin

:wq
source /etc/profile
檢查：
mvn -version
  1. 配置ant：先解壓，再配置環境變數，和JDK配置一樣
vi /etc/profile

#MAVEN_HOME
export MAVEN_HOME=/opt/install/apache-ant-1.9.9
export PATH=$PATH:$MAVEN_HOME/bin

:wq
source /etc/profile
ant -version
  1. 安裝glibc-headers、g++、make和cmake,openssl庫與ncurses-devel庫。yum安裝即可
yum -y install glibc-headers
yum -y install gcc-c++
yum -y install make
yum -y install cmake
yum -y install openssl-devel
yum -y install ncurses-devel
  1. 安裝protobuf,先解壓，再進入目錄：
cd /opt/install/protobuf-2.5.0/
./configure
make
make check
make install
ldconfig

編輯配置檔案
vi /etc/profile

#LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/install/protobuf-2.5.0
export PATH=$PATH:$LD_LIBRARY_PATH

:wq
source /etc/profile
protoc --version
  1. 編譯原始碼
tar -zxvf hadoop-2.6.0-src.tar.gz
進入cd /opt/install/hadoop-2.7.2-src/
這裡需要修改一下我們JDK對應的版本：
vi pom.xml

	  <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-enforcer-plugin</artifactId>
          <version>1.3.1</version>
          <configuration>
            <rules>
              <requireMavenVersion>
                <version>[3.0.2,)</version>
              </requireMavenVersion>
              <requireJavaVersion>
                <!- 在這裡修改對應版本 ->
                <version>1.7</version>
              </requireJavaVersion>
            </rules>
          </configuration>
        </plugin>
        
        
mvn package -Pdist,native -DskipTests -Dtar

接下來就會開始原始碼編譯，但這個過程會很漫長。我自己是編譯了差不多四個小時，這都取決於網速和配置等因素。

最後，編譯成功後我們要的安裝包在：/opt/install/hadoop-2.7.2-src/hadoop-dist/target/下

相關文章