centos6下安裝部署hadoop2.2

環境準備
1、作業系統:centos6.0 64位
2、hadoop版本：hahadoop-2.2.0

安裝和配置步驟具體如下：
1、主機和ip分配如下
     ip地址                  主機名               用途
     192.168.1.112      hadoop1            namenode
     192.168.1.113      hadoop2            datanode
     192.168.1.114      hadoop3            datanode
     192.168.1.115      hadoop4            datanode
2、四臺主機主機名修改如下，僅以hadoop1為例
    1) [root@hadoop1 ~]# hostname hadoop1
    2) [root@hadoop1 ~]# vi /etc/sysconfig/network,修改hostname屬性值儲存
3、四臺主機都需要安裝jdk
4、關閉防火牆，切換到root使用者執行如下命令 chkconfig iptables off
5、每臺主機配置/etc/hosts 增加ip地址解析

hadoop1 192.168.1.112

hadoop2 192.168.1.113

hadoop3 192.168.1.114

    hadoop4 192.168.1.115
5、配置namenode無密碼訪問datanode
   1) 在namenode機器上，在hadoop使用者下執行下面命令
     ssh-keygen -t rsa
     遇到所有選項回車預設值即可
   2) 匯入公鑰到本機認證檔案
      cat ~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys
   3) 匯入公鑰到其他datanode節點認證檔案

scp ～/.ssh/authorized_keys hadoop@192.168.1.113:/home/hadoop/.ssh/authorized_keys

scp ～/.ssh/authorized_keys hadoop@192.168.1.114:/home/hadoop/.ssh/authorized_keys

    scp ～/.ssh/authorized_keys hadoop@192.168.1.115:/home/hadoop/.ssh/authorized_keys
    以上過程由於是第一次傳輸訪問，系統會提示輸入hadoop使用者的密碼，輸入密碼即可。
     4) 驗證是否能夠無密碼登陸到datanode節點
      [hadoop@hadoop1 ~]$ ssh 192.168.1.113
      如果沒有密碼提示直接登陸顯示如下
        [hadoop@hadoop2 ~]$
       無密碼登陸驗證通過，訪問其他datanode節點類似
6、安裝hadoop2.2
   1）解壓縮hadoop-2.2.0.tar.gz
       tar -zxf hadoop-2.2.0.tar.gz
       預設解壓縮到當前目錄下面，這裡解壓縮/home/hadoop/目錄下面
   2) 修改hadoop配置檔案
       開啟hadoop-2.2.0/etc/hadoop，修改裡面的配置檔案

      a) hadoop-env.sh，找到裡面的JAVA_HOME,修改為實際地址
      b) yarn-env.sh ，同樣找到裡面的JAVA_HOME,修改為實際路徑
      c) slave 這個檔案配置所有datanode節點，以便namenode搜尋，本例配置如下
         hadoop2
         hadoop3
         hadoop4
      d) core-site.xml

fs.defaultFS

hdfs://hadoop1:9000

io.file.buffer.size

131072

hadoop.tmp.dir

/home/hadoop/hadoopp-2.2.0/mytmp

A base for other temporarydirectories.

hadoop.proxyuser.root.hosts

hadoop1

hadoop.proxyuser.root.groups

*

e) hdfs-site.xml

dfs.namenode.name.dir

/home/hadoop/name

true

dfs.datanode.data.dir

/home/hadoop/data

true

dfs.replication

3

dfs.permissions

false

f) mapred-site.xml

mapreduce.framework.name

yarn

mapreduce.jobhistory.address

hadoop1:10020

mapreduce.jobhistory.webapp.address

hadoop1:19888

mapreduce.jobhistory.intermediate-done-dir

/mr-history/tmp

mapreduce.jobhistory.done-dir

/mr-history/done

g) yarn-site.xml

yarn.resourcemanager.address

hadoop1:18040

yarn.resourcemanager.scheduler.address

hadoop1:18030

yarn.resourcemanager.resource-tracker.address

hadoop1:18025

yarn.resourcemanager.admin.address

hadoop1:18041

yarn.resourcemanager.webapp.address

hadoop1:8088

yarn.nodemanager.local-dirs

/home/hadoop/mynode/my

yarn.nodemanager.log-dirs

/home/hadoop/mynode/logs

yarn.nodemanager.log.retain-seconds

10800

yarn.nodemanager.remote-app-log-dir

/logs

yarn.nodemanager.remote-app-log-dir-suffix

logs

yarn.log-aggregation.retain-seconds

-1

yarn.log-aggregation.retain-check-interval-seconds

-1

yarn.nodemanager.aux-services

mapreduce_shuffle

3) 將上述檔案配置好後，將hadoop-2.2.0檔案複製到其餘datanode機器上的相同路徑下
    修改/etc/profile檔案設定hadoop環境變數，切換到root使用者
     在檔案的最後增加如下配置

#hadoop variable settings

export HADOOP_HOME=/home/hadoop/hadoop-2.2.0

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib

    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
    增加之後儲存
    最後兩行特殊說明下，有的文章中遺漏掉這部分配置，最後在啟動hadoop2.2時報了下面的錯誤

Hadoop 2.2.0 - warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard.


	Unable to load native-hadoop library for your platform... using builtin-java classes where applicable



	Starting namenodes on [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.



	It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.



	localhost]



	sed: -e expression #1, char 6: unknown option to `s'



	HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Name or service not known



	64-Bit: ssh: Could not resolve hostname 64-Bit: Name or service not known



	Java: ssh: Could not resolve hostname Java: Name or service not known



	Server: ssh: Could not resolve hostname Server: Name or service not known



	VM: ssh: Could not resolve hostname VM: Name or service not known

配置完成之後需要重啟電腦，所有datanode節點也需要對環境變數增加上面配置，配置完成之後重啟電腦
7、hadoop的啟動與關閉
   1）hadoop namenode的初始化,只需要第一次的時候初始化，之後就不需要了
       cd /home/hadoop/hadoop-2.2.0/bin
        hdfs namenode -format
   2）啟動
       啟動：在namenode機器上，進入/home/myhadoop/sbin
       執行指令碼start-all.sh
   3) 關閉
       在namenode節點上，進入/home/myhadoop/sbin
      stop-all.sh
8、web介面地址

啟動hadoop後，在瀏覽器中輸入地址檢視

http://hadoop1:50070

http://hadoop1:8088

http://hadoop1:19888

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/7483882/viewspace-1069062/，如需轉載，請註明出處，否則將追究法律責任。

centos6下安裝部署hadoop2.2

相關文章