【實驗】Hadoop-2.7.2+zookeeper-3.4.6完全分散式環境搭建(HDFS、YARN HA)

hackeruncle發表於2016-02-25
Hadoop分散式Yarn

Hadoop-2.7.2+Zookeeper-3.4.6完全分散式環境搭建

一.版本

元件名

版本

說明

JRE

java version "1.7.0_67"

Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

 

Hadoop

hadoop-2.7.2.tar.gz

主程式包

Zookeeper

zookeeper-3.4.6.tar.gz

熱切,Yarn 儲存資料使用的協調服務

二.主機規劃

IP

Host 及安裝軟體

部署模組

程式

172.16.101.55

sht-sgmhadoopnn-01

hadoop

NameNode

ResourceManager

NameNode

DFSZKFailoverController

ResourceManager

172.16.101.56

sht-sgmhadoopnn-02

hadoop

NameNode

ResourceManager

NameNode

DFSZKFailoverController

ResourceManager

172.16.101.58

sht-sgmhadoopdn-01

hadoop、zookeeper

DataNode

NodeManager

Zookeeper

DataNode

NodeManager

JournalNode

QuorumPeerMain

172.16.101.59

sht-sgmhadoopdn-02

Hadoop、zookeeper

DataNode

NodeManager

Zookeeper

DataNode

NodeManager

JournalNode

QuorumPeerMain

172.16.101.60

sht-sgmhadoopdn-03

Hadoop、zookeeper

DataNode

NodeManager

Zookeeper

DataNode

NodeManager

JournalNode

QuorumPeerMain

三.目錄規劃

名稱

路徑

$HADOOP_HOME

/hadoop/hadoop-2.7.2

Data

$ HADOOP_HOME/data

Log

$ HADOOP_HOME/logs

四.常用指令碼及命令

1.啟動叢集

start-dfs.sh

start-yarn.sh

2.關閉叢集

stop-yarn.sh

stop-dfs.sh

3.監控叢集

hdfs dfsadmin -report

4.單個程式啟動/關閉

hadoop-daemon.sh start|stop namenode|datanode| journalnode

yarn-daemon.sh start |stop resourcemanager|nodemanager

http://blog.chinaunix.net/uid-25723371-id-4943894.html

五.環境準備

1 .設定ip地址(5臺)

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
  2. DEVICE="eth0"
  3. BOOTPROTO="static"
  4. DNS1="172.16.101.63"
  5. DNS2="172.16.101.64"
  6. GATEWAY="172.16.101.1"
  7. HWADDR="00:50:56:82:50:1E"
  8. IPADDR="172.16.101.55"
  9. NETMASK="255.255.255.0"
  10. NM_CONTROLLED="yes"
  11. ONBOOT="yes"
  12. TYPE="Ethernet"
  13. UUID="257c075f-6c6a-47ef-a025-e625367cbd9c"


執行命令: service network restart

驗證:ifconfig

2 .關閉防火牆(5臺)

執行命:service iptables stop

驗證:service iptables status

3.關閉防火牆的自動執行(5臺)

執行命令:chkconfig iptables off

驗證:chkconfig --list | grep iptables

4 設定主機名(5臺)

執行命令
(1)hostname sht-sgmhadoopnn-01

(2)vi /etc/sysconfig/network

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 ~]# vi /etc/sysconfig/network
  2. NETWORKING=yes
  3. HOSTNAME=sht-sgmhadoopnn-01.telenav.cn
  4. GATEWAY=172.16.101.1

5 ip與hostname繫結(5臺)

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 ~]# vi /etc/hosts
  2. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
  3. ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
  4. 172.16.101.55 sht-sgmhadoopnn-01.telenav.cn sht-sgmhadoopnn-01
  5. 172.16.101.56 sht-sgmhadoopnn-02.telenav.cn sht-sgmhadoopnn-02
  6. 172.16.101.58 sht-sgmhadoopdn-01.telenav.cn sht-sgmhadoopdn-01
  7. 172.16.101.59 sht-sgmhadoopdn-02.telenav.cn sht-sgmhadoopdn-02
  8. 172.16.101.60 sht-sgmhadoopdn-03.telenav.cn sht-sgmhadoopdn-03
  9. 驗證:ping sht-sgmhadoopnn-01

6. 設定5臺machines,SSH互相通訊
http://blog.itpub.net/30089851/viewspace-1992210/

7 .安裝JDK(5臺)

點選(此處)摺疊或開啟

  1. (1)執行命令
  2. [root@sht-sgmhadoopnn-01 ~]# cd /usr/java
  3. [root@sht-sgmhadoopnn-01 java]# cp /tmp/jdk-7u67-linux-x64.gz ./
  4. [root@sht-sgmhadoopnn-01 java]# tar -xzvf jdk-7u67-linux-x64.gz
  5. (2)vi /etc/profile 增加內容如下:
  6. export JAVA_HOME=/usr/java/jdk1.7.0_67
  7. export HADOOP_HOME=/hadoop/hadoop-2.7.2
  8. export ZOOKEEPER_HOME=/hadoop/zookeeper
  9. export PATH=.:$HADOOP_HOME/bin:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$PATH
  10. #先把HADOOP_HOME, ZOOKEEPER_HOME配置了
  11. #本次實驗機器已經配置好了jdk1.7.0_67-cloudera
  12. (3)執行 source /etc/profile
  13. (4)驗證:java –version

8.建立資料夾(5臺)

mkdir /hadoop

六.安裝Zookeeper

sht-sgmhadoopdn-01/02/03

1.下載解壓zookeeper-3.4.6.tar.gz

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopdn-01 tmp]# wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
  2. [root@sht-sgmhadoopdn-02 tmp]# wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
  3. [root@sht-sgmhadoopdn-03 tmp]# wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
  4. [root@sht-sgmhadoopdn-01 tmp]# tar -xvf zookeeper-3.4.6.tar.gz
  5. [root@sht-sgmhadoopdn-02 tmp]# tar -xvf zookeeper-3.4.6.tar.gz
  6. [root@sht-sgmhadoopdn-03 tmp]# tar -xvf zookeeper-3.4.6.tar.gz
  7. [root@sht-sgmhadoopdn-01 tmp]# mv zookeeper-3.4.6 /hadoop/zookeeper
  8. [root@sht-sgmhadoopdn-02 tmp]# mv zookeeper-3.4.6 /hadoop/zookeeper
  9. [root@sht-sgmhadoopdn-03 tmp]# mv zookeeper-3.4.6 /hadoop/zookeeper

2.修改配置

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopdn-01 tmp]# cd /hadoop/zookeeper/conf
  2. [root@sht-sgmhadoopdn-01 conf]# cp zoo_sample.cfg zoo.cfg
  3. [root@sht-sgmhadoopdn-01 conf]# vi zoo.cfg
  4. 修改dataDir
  5. dataDir=/hadoop/zookeeper/data
  6. 新增下面三行
  7. server.1=sht-sgmhadoopdn-01:2888:3888
  8. server.2=sht-sgmhadoopdn-02:2888:3888
  9. server.3=sht-sgmhadoopdn-03:2888:3888
  10. [root@sht-sgmhadoopdn-01 conf]# cd ../
  11. [root@sht-sgmhadoopdn-01 zookeeper]# mkdir data
  12. [root@sht-sgmhadoopdn-01 zookeeper]# touch data/myid
  13. [root@sht-sgmhadoopdn-01 zookeeper]# echo 1 > data/myid
  14. [root@sht-sgmhadoopdn-01 zookeeper]# more data/myid
  15. 1
  16. ## sht-sgmhadoopdn-02/03,也修改配置,就如下不同
  17. [root@sht-sgmhadoopdn-02 zookeeper]# echo 2 > data/myid
  18. [root@sht-sgmhadoopdn-03 zookeeper]# echo 3 > data/myid

七.安裝Hadoop(HDFS HA+YARN HA)

#step3~7,用SecureCRT ssh 到 linux的環境中,假如copy 內容從window 到 linux 中,中文亂碼,請參照修改http://www.cnblogs.com/qi09/archive/2013/02/05/2892922.html

1.下載解壓hadoop-2.7.2.tar.gz

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopdn-01 tmp]# cd /hadoop/zookeeper/conf
  2. [root@sht-sgmhadoopdn-01 conf]# cp zoo_sample.cfg zoo.cfg
  3. [root@sht-sgmhadoopdn-01 conf]# vi zoo.cfg
  4. 修改dataDir
  5. dataDir=/hadoop/zookeeper/data
  6. 新增下面三行
  7. server.1=sht-sgmhadoopdn-01:2888:3888
  8. server.2=sht-sgmhadoopdn-02:2888:3888
  9. server.3=sht-sgmhadoopdn-03:2888:3888
  10. [root@sht-sgmhadoopdn-01 conf]# cd ../
  11. [root@sht-sgmhadoopdn-01 zookeeper]# mkdir data
  12. [root@sht-sgmhadoopdn-01 zookeeper]# touch data/myid
  13. [root@sht-sgmhadoopdn-01 zookeeper]# echo 1 > data/myid
  14. [root@sht-sgmhadoopdn-01 zookeeper]# more data/myid
  15. 1
  16. ## sht-sgmhadoopdn-02/03,也修改配置,就如下不同
  17. [root@sht-sgmhadoopdn-02 zookeeper]# echo 2 > data/myid
  18. [root@sht-sgmhadoopdn-03 zookeeper]# echo 3 > data/myid

2.修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh

export JAVA_HOME="/usr/java/jdk1.7.0_67-cloudera"

3.修改$HADOOP_HOME/etc/hadoop/core-site.xml

點選(此處)摺疊或開啟

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <configuration>
  4.     <!--Yarn 需要使用 fs.defaultFS 指定NameNode URI -->
  5.     <property>
  6.         <name>fs.defaultFS</name>
  7.         <value>hdfs://mycluster</value>
  8.     </property>
  9.     <!--HDFS超級使用者 -->
  10.     <property>
  11.         <name>dfs.permissions.superusergroup</name>
  12.         <value>root</value>
  13.     </property>
  14.     <!--==============================Trash機制======================================= -->
  15.     <property>
  16.         <!--多長時間建立CheckPoint NameNode截點上執行的CheckPointer 從Current資料夾建立CheckPoint;預設：0 由fs.trash.interval項指定 -->
  17.         <name>fs.trash.checkpoint.interval</name>
  18.         <value>0</value>
  19.     </property>
  20.     <property>
  21.         <!--多少分鐘.Trash下的CheckPoint目錄會被刪除,該配置伺服器設定優先順序大於客戶端，預設：0 不刪除 -->
  22.         <name>fs.trash.interval</name>
  23.         <value>1440</value>
  24.     </property>
  25. </configuration>


4.修改$HADOOP_HOME/etc/hadoop/hdfs-site.xml


點選(此處)摺疊或開啟

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <configuration>
  4.     <!--開啟web hdfs -->
  5.     <property>
  6.         <name>dfs.webhdfs.enabled</name>
  7.         <value>true</value>
  8.     </property>
  9.     <property>
  10.         <name>dfs.namenode.name.dir</name>
  11.         <value>/hadoop/hadoop-2.7.2/data/dfs/name</value>
  12.         <description> namenode 存放name table(fsimage)本地目錄（需要修改）</description>
  13.     </property>
  14.     <property>
  15.         <name>dfs.namenode.edits.dir</name>
  16.         <value>${dfs.namenode.name.dir}</value>
  17.         <description>namenode粗放 transaction file(edits)本地目錄（需要修改）</description>
  18.     </property>
  19.     <property>
  20.         <name>dfs.datanode.data.dir</name>
  21.         <value>/hadoop/hadoop-2.7.2/data/dfs/data</value>
  22.         <description>datanode存放block本地目錄（需要修改）</description>
  23.     </property>
  24.     <property>
  25.         <name>dfs.replication</name>
  26.         <value>3</value>
  27.     </property>
  28.     <!-- 塊大小 （預設） -->
  29.     <property>
  30.         <name>dfs.blocksize</name>
  31.         <value>268435456</value>
  32.     </property>
  33.     <!--======================================================================= -->
  34.     <!--HDFS高可用配置 -->
  35.     <!--nameservices邏輯名 -->
  36.     <property>
  37.         <name>dfs.nameservices</name>
  38.         <value>mycluster</value>
  39.     </property>
  40.     <property>
  41.         <!--設定NameNode IDs 此版本最大隻支援兩個NameNode -->
  42.         <name>dfs.ha.namenodes.mycluster</name>
  43.         <value>nn1,nn2</value>
  44.     </property>

  46.     <!-- Hdfs HA: dfs.namenode.rpc-address.[nameservice ID] rpc 通訊地址 -->
  47.     <property>
  48.         <name>dfs.namenode.rpc-address.mycluster.nn1</name>
  49.         <value>sht-sgmhadoopnn-01:8020</value>
  50.     </property>
  51.     <property>
  52.         <name>dfs.namenode.rpc-address.mycluster.nn2</name>
  53.         <value>sht-sgmhadoopnn-02:8020</value>
  54.     </property>

  56.     <!-- Hdfs HA: dfs.namenode.http-address.[nameservice ID] http 通訊地址 -->
  57.     <property>
  58.         <name>dfs.namenode.http-address.mycluster.nn1</name>
  59.         <value>sht-sgmhadoopnn-01:50070</value>
  60.     </property>
  61.     <property>
  62.         <name>dfs.namenode.http-address.mycluster.nn2</name>
  63.         <value>sht-sgmhadoopnn-02:50070</value>
  64.     </property>

  66.     <!--==================Namenode editlog同步 ============================================ -->
  67.     <!--保證資料恢復 -->
  68.     <property>
  69.         <name>dfs.journalnode.http-address</name>
  70.         <value>0.0.0.0:8480</value>
  71.     </property>
  72.     <property>
  73.         <name>dfs.journalnode.rpc-address</name>
  74.         <value>0.0.0.0:8485</value>
  75.     </property>
  76.     <property>
  77.         <!--設定JournalNode伺服器地址，QuorumJournalManager 用於儲存editlog -->
  78.         <!--格式：qjournal://<host1:port1>;<host2:port2>;<host3:port3>/<journalId> 埠同journalnode.rpc-address -->
  79.         <name>dfs.namenode.shared.edits.dir</name>
  80.         <value>qjournal://sht-sgmhadoopdn-01:8485;sht-sgmhadoopdn-02:8485;sht-sgmhadoopdn-03:8485/mycluster</value>
  81.     </property>

  83.     <property>
  84.         <!--JournalNode存放資料地址 -->
  85.         <name>dfs.journalnode.edits.dir</name>
  86.         <value>/hadoop/hadoop-2.7.2/data/dfs/jn</value>
  87.     </property>
  88.     <!--==================DataNode editlog同步 ============================================ -->
  89.     <property>
  90.         <!--DataNode,Client連線Namenode識別選擇Active NameNode策略 -->
  91.         <name>dfs.client.failover.proxy.provider.mycluster</name>
  92.         <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  93.     </property>
  94.     <!--==================Namenode fencing：=============================================== -->
  95.     <!--Failover後防止停掉的Namenode啟動，造成兩個服務 -->
  96.     <property>
  97.         <name>dfs.ha.fencing.methods</name>
  98.         <value>sshfence</value>
  99.     </property>
  100.     <property>
  101.         <name>dfs.ha.fencing.ssh.private-key-files</name>
  102.         <value>/root/.ssh/id_rsa</value>
  103.     </property>
  104.     <property>
  105.         <!--多少milliseconds 認為fencing失敗 -->
  106.         <name>dfs.ha.fencing.ssh.connect-timeout</name>
  107.         <value>30000</value>
  108.     </property>

  110.     <!--==================NameNode auto failover base ZKFC and Zookeeper====================== -->
  111.     <!--開啟基於Zookeeper及ZKFC程式的自動備援設定,監視程式是否死掉 -->
  112.     <property>
  113.         <name>dfs.ha.automatic-failover.enabled</name>
  114.         <value>true</value>
  115.     </property>
  116.     <property>
  117.         <name>ha.zookeeper.quorum</name>
  118.         <value>sht-sgmhadoopdn-01:2181,sht-sgmhadoopdn-02:2181,sht-sgmhadoopdn-03:2181</value>
  119.     </property>
  120.     <property>
  121.         <!--指定ZooKeeper超時間隔，單位毫秒 -->
  122.         <name>ha.zookeeper.session-timeout.ms</name>
  123.         <value>2000</value>
  124.     </property>
  125. </configuration>

5.修改$HADOOP_HOME/etc/hadoop/yarn-env.sh

#Yarn Daemon Options

#export YARN_RESOURCEMANAGER_OPTS

#export YARN_NODEMANAGER_OPTS

#export YARN_PROXYSERVER_OPTS

#export HADOOP_JOB_HISTORYSERVER_OPTS

#Yarn Logs

export YARN_LOG_DIR="/hadoop/hadoop-2.7.2/logs"

6.修改$HADOOP_HOEM/etc/hadoop/mapred-site.xml

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 hadoop]# cp mapred-site.xml.template mapred-site.xml
  2. [root@sht-sgmhadoopnn-01 hadoop]# vi mapred-site.xml
  3. <configuration>
  4.     <!-- 配置 MapReduce Applications -->
  5.     <property>
  6.         <name>mapreduce.framework.name</name>
  7.         <value>yarn</value>
  8.     </property>
  9.     <!-- JobHistory Server ============================================================== -->
  10.     <!-- 配置 MapReduce JobHistory Server 地址 ，預設: 0.0.0.0:10020 -->
  11.     <property>
  12.         <name>mapreduce.jobhistory.address</name>
  13.         <value>sht-sgmhadoopnn-01:10020</value>
  14.     </property>
  15.     <!-- 配置 MapReduce JobHistory Server web ui 地址， 預設: 0.0.0.0:19888 -->
  16.     <property>
  17.         <name>mapreduce.jobhistory.webapp.address</name>
  18.         <value>sht-sgmhadoopnn-01:19888</value>
  19.     </property>
  20. </configuration>

7.修改$HADOOP_HOME/etc/hadoop/yarn-site.xml

點選(此處)摺疊或開啟

  1. <configuration>
  2.     <!-- nodemanager 配置 ================================================= -->
  3.     <property>
  4.         <name>yarn.nodemanager.aux-services</name>
  5.         <value>mapreduce_shuffle</value>
  6.     </property>
  7.     <property>
  8.         <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  9.         <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  10.     </property>
  11.     <property>
  12.         <description>Address where the localizer IPC is.</description>
  13.         <name>yarn.nodemanager.localizer.address</name>
  14.         <value>0.0.0.0:23344</value>
  15.     </property>
  16.     <property>
  17.         <description>NM Webapp address.</description>
  18.         <name>yarn.nodemanager.webapp.address</name>
  19.         <value>0.0.0.0:23999</value>
  20.     </property>

  22.     <!-- HA 配置 =============================================================== -->
  23.     <!-- Resource Manager Configs -->
  24.     <property>
  25.         <name>yarn.resourcemanager.connect.retry-interval.ms</name>
  26.         <value>2000</value>
  27.     </property>
  28.     <property>
  29.         <name>yarn.resourcemanager.ha.enabled</name>
  30.         <value>true</value>
  31.     </property>
  32.     <property>
  33.         <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
  34.         <value>true</value>
  35.     </property>
  36.     <!-- 使嵌入式自動故障轉移。HA環境啟動，與 ZKRMStateStore 配合 處理fencing -->
  37.     <property>
  38.         <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
  39.         <value>true</value>
  40.     </property>
  41.     <!-- 叢集名稱，確保HA選舉時對應的叢集 -->
  42.     <property>
  43.         <name>yarn.resourcemanager.cluster-id</name>
  44.         <value>yarn-cluster</value>
  45.     </property>
  46.     <property>
  47.         <name>yarn.resourcemanager.ha.rm-ids</name>
  48.         <value>rm1,rm2</value>
  49.     </property>
  50.     <!--這裡RM主備結點需要單獨指定,（可選）
  51.     <property>
  52.         <name>yarn.resourcemanager.ha.id</name>
  53.         <value>rm2</value>
  54. </property>
  55.  -->
  56.     <property>
  57.         <name>yarn.resourcemanager.scheduler.class</name>
  58.         <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
  59.     </property>
  60.     <property>
  61.         <name>yarn.resourcemanager.recovery.enabled</name>
  62.         <value>true</value>
  63.     </property>
  64.     <property>
  65.         <name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
  66.         <value>5000</value>
  67.     </property>
  68.     <!-- ZKRMStateStore 配置 -->
  69.     <property>
  70.         <name>yarn.resourcemanager.store.class</name>
  71.         <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
  72.     </property>
  73.     <property>
  74.         <name>yarn.resourcemanager.zk-address</name>
  75.         <value>sht-sgmhadoopdn-01:2181,sht-sgmhadoopdn-02:2181,sht-sgmhadoopdn-03:2181</value>
  76.     </property>
  77.     <property>
  78.         <name>yarn.resourcemanager.zk.state-store.address</name>
  79.         <value>sht-sgmhadoopdn-01:2181,sht-sgmhadoopdn-02:2181,sht-sgmhadoopdn-03:2181</value>
  80.     </property>
  81.     <!-- Client訪問RM的RPC地址 (applications manager interface) -->
  82.     <property>
  83.         <name>yarn.resourcemanager.address.rm1</name>
  84.         <value>sht-sgmhadoopnn-01:23140</value>
  85.     </property>
  86.     <property>
  87.         <name>yarn.resourcemanager.address.rm2</name>
  88.         <value>sht-sgmhadoopnn-02:23140</value>
  89.     </property>
  90.     <!-- AM訪問RM的RPC地址(scheduler interface) -->
  91.     <property>
  92.         <name>yarn.resourcemanager.scheduler.address.rm1</name>
  93.         <value>sht-sgmhadoopnn-01:23130</value>
  94.     </property>
  95.     <property>
  96.         <name>yarn.resourcemanager.scheduler.address.rm2</name>
  97.         <value>sht-sgmhadoopnn-02:23130</value>
  98.     </property>
  99.     <!-- RM admin interface -->
  100.     <property>
  101.         <name>yarn.resourcemanager.admin.address.rm1</name>
  102.         <value>sht-sgmhadoopnn-01:23141</value>
  103.     </property>
  104.     <property>
  105.         <name>yarn.resourcemanager.admin.address.rm2</name>
  106.         <value>sht-sgmhadoopnn-02:23141</value>
  107.     </property>
  108.     <!--NM訪問RM的RPC埠 -->
  109.     <property>
  110.         <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
  111.         <value>sht-sgmhadoopnn-01:23125</value>
  112.     </property>
  113.     <property>
  114.         <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
  115.         <value>sht-sgmhadoopnn-02:23125</value>
  116.     </property>
  117.     <!-- RM web application 地址 -->
  118.     <property>
  119.         <name>yarn.resourcemanager.webapp.address.rm1</name>
  120.         <value>sht-sgmhadoopnn-01:8088</value>
  121.     </property>
  122.     <property>
  123.         <name>yarn.resourcemanager.webapp.address.rm2</name>
  124.         <value>sht-sgmhadoopnn-02:8088</value>
  125.     </property>
  126.     <property>
  127.         <name>yarn.resourcemanager.webapp.https.address.rm1</name>
  128.         <value>sht-sgmhadoopnn-01:23189</value>
  129.     </property>
  130.     <property>
  131.         <name>yarn.resourcemanager.webapp.https.address.rm2</name>
  132.         <value>sht-sgmhadoopnn-02:23189</value>
  133.     </property>
  134. </configuration>

8.修改slaves

[root@sht-sgmhadoopnn-01 hadoop]# vi slaves

sht-sgmhadoopdn-01

sht-sgmhadoopdn-02

sht-sgmhadoopdn-03

9.分發資料夾

[root@sht-sgmhadoopnn-01 hadoop]# scp -r hadoop-2.7.2 root@sht-sgmhadoopnn-02:/hadoop

[root@sht-sgmhadoopnn-01 hadoop]# scp -r hadoop-2.7.2 root@sht-sgmhadoopdn-01:/hadoop

[root@sht-sgmhadoopnn-01 hadoop]# scp -r hadoop-2.7.2 root@sht-sgmhadoopdn-02:/hadoop

[root@sht-sgmhadoopnn-01 hadoop]# scp -r hadoop-2.7.2 root@sht-sgmhadoopdn-03:/hadoop

八.啟動叢集

另外一種啟動方式:

1.啟動zookeeper

點選(此處)摺疊或開啟

  1. command: ./zkServer.sh start|stop|status
  2. [root@sht-sgmhadoopdn-01 bin]# ./zkServer.sh start
  3. JMX enabled by default
  4. Using config: /hadoop/zookeeper/bin/../conf/zoo.cfg
  5. Starting zookeeper ... STARTED
  6. [root@sht-sgmhadoopdn-01 bin]# jps
  7. 2073 QuorumPeerMain
  8. 2106 Jps
  9. [root@sht-sgmhadoopdn-02 bin]# ./zkServer.sh start
  10. JMX enabled by default
  11. Using config: /hadoop/zookeeper/bin/../conf/zoo.cfg
  12. Starting zookeeper ... STARTED
  13. [root@sht-sgmhadoopdn-02 bin]# jps
  14. 2073 QuorumPeerMain
  15. 2106 Jps
  16. [root@sht-sgmhadoopdn-03 bin]# ./zkServer.sh start
  17. JMX enabled by default
  18. Using config: /hadoop/zookeeper/bin/../conf/zoo.cfg
  19. Starting zookeeper ... STARTED
  20. [root@sht-sgmhadoopdn-03 bin]# jps
  21. 2073 QuorumPeerMain
  22. 2106 Jps

2.啟動hadoop(HDFS+YARN)

a.格式化前,先在journalnode 節點機器上先啟動JournalNode程式

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopdn-01 ~]# cd /hadoop/hadoop-2.7.2/sbin
  2. [root@sht-sgmhadoopdn-01 sbin]# hadoop-daemon.sh start journalnode
  3. starting journalnode, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-journalnode-sht-sgmhadoopdn-03.telenav.cn.out
  4. [root@sht-sgmhadoopdn-03 sbin]# jps
  5. 16722 JournalNode
  6. 16775 Jps
  7. 15519 QuorumPeerMain
  8. [root@sht-sgmhadoopdn-02 ~]# cd /hadoop/hadoop-2.7.2/sbin
  9. [root@sht-sgmhadoopdn-02 sbin]# hadoop-daemon.sh start journalnode
  10. starting journalnode, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-journalnode-sht-sgmhadoopdn-03.telenav.cn.out
  11. [root@sht-sgmhadoopdn-03 sbin]# jps
  12. 16722 JournalNode
  13. 16775 Jps
  14. 15519 QuorumPeerMain
  15. [root@sht-sgmhadoopdn-03 ~]# cd /hadoop/hadoop-2.7.2/sbin
  16. [root@sht-sgmhadoopdn-03 sbin]# hadoop-daemon.sh start journalnode
  17. starting journalnode, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-journalnode-sht-sgmhadoopdn-03.telenav.cn.out
  18. [root@sht-sgmhadoopdn-03 sbin]# jps
  19. 16722 JournalNode
  20. 16775 Jps
  21. 15519 QuorumPeerMain

b.NameNode格式化

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 bin]# hadoop namenode -format
  2. 16/02/25 14:05:04 INFO namenode.NameNode: STARTUP_MSG:
  3. /************************************************************
  4. STARTUP_MSG: Starting NameNode
  5. STARTUP_MSG: host = sht-sgmhadoopnn-01.telenav.cn/172.16.101.55
  6. STARTUP_MSG: args = [-format]
  7. STARTUP_MSG: version = 2.7.2
  8. STARTUP_MSG: classpath =
  9. ……………..
  10. ………………
  11. 16/02/25 14:05:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
  12. 16/02/25 14:05:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
  13. 16/02/25 14:05:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
  14. 16/02/25 14:05:07 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
  15. 16/02/25 14:05:07 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
  16. 16/02/25 14:05:07 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
  17. 16/02/25 14:05:07 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
  18. 16/02/25 14:05:07 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
  19. 16/02/25 14:05:07 INFO util.GSet: Computing capacity for map NameNodeRetryCache
  20. 16/02/25 14:05:07 INFO util.GSet: VM type = 64-bit
  21. 16/02/25 14:05:07 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
  22. 16/02/25 14:05:07 INFO util.GSet: capacity = 2^15 = 32768 entries
  23. 16/02/25 14:05:08 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1182930464-172.16.101.55-1456380308394
  24. 16/02/25 14:05:08 INFO common.Storage: Storage directory /hadoop/hadoop-2.7.2/data/dfs/name has been successfully formatted.
  25. 16/02/25 14:05:08 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
  26. 16/02/25 14:05:08 INFO util.ExitUtil: Exiting with status 0
  27. 16/02/25 14:05:08 INFO namenode.NameNode: SHUTDOWN_MSG:
  28. /************************************************************
  29. SHUTDOWN_MSG: Shutting down NameNode at sht-sgmhadoopnn-01.telenav.cn/172.16.101.55
  30. ************************************************************/

c.同步NameNode後設資料

點選(此處)摺疊或開啟

  1. 同步sht-sgmhadoopnn-01 後設資料到sht-sgmhadoopnn-02
  2. 主要是：dfs.namenode.name.dir，dfs.namenode.edits.dir還應該確保共享儲存目錄下(dfs.namenode.shared.edits.dir ) 包含NameNode 所有的後設資料。
  3. [root@sht-sgmhadoopnn-01 hadoop-2.7.2]# pwd
  4. /hadoop/hadoop-2.7.2
  5. [root@sht-sgmhadoopnn-01 hadoop-2.7.2]# scp -r data/ root@sht-sgmhadoopnn-02:/hadoop/hadoop-2.7.2
  6. seen_txid 100% 2 0.0KB/s 00:00
  7. fsimage_0000000000000000000 100% 351 0.3KB/s 00:00
  8. fsimage_0000000000000000000.md5 100% 62 0.1KB/s 00:00
  9. VERSION 100% 205 0.2KB/s 00:00

d.初始化ZFCK

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 bin]# hdfs zkfc -formatZK
  2. ……………..
  3. ……………..
  4. 16/02/25 14:14:41 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
  5. 16/02/25 14:14:41 INFO zookeeper.ZooKeeper: Client environment:user.dir=/hadoop/hadoop-2.7.2/bin
  6. 16/02/25 14:14:41 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=sht-sgmhadoopdn-01:2181,sht-sgmhadoopdn-02:2181,sht-sgmhadoopdn-03:2181 sessionTimeout=2000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@5f4298a5
  7. 16/02/25 14:14:41 INFO zookeeper.ClientCnxn: Opening socket connection to server sht-sgmhadoopdn-01.telenav.cn/172.16.101.58:2181. Will not attempt to authenticate using SASL (unknown error)
  8. 16/02/25 14:14:41 INFO zookeeper.ClientCnxn: Socket connection established to sht-sgmhadoopdn-01.telenav.cn/172.16.101.58:2181, initiating session
  9. 16/02/25 14:14:42 INFO zookeeper.ClientCnxn: Session establishment complete on server sht-sgmhadoopdn-01.telenav.cn/172.16.101.58:2181, sessionid = 0x15316c965750000, negotiated timeout = 4000
  10. 16/02/25 14:14:42 INFO ha.ActiveStandbyElector: Session connected.
  11. 16/02/25 14:14:42 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/mycluster in ZK.
  12. 16/02/25 14:14:42 INFO zookeeper.ClientCnxn: EventThread shut down
  13. 16/02/25 14:14:42 INFO zookeeper.ZooKeeper: Session: 0x15316c965750000 closed

e.啟動HDFS 系統

叢集啟動,在sht-sgmhadoopnn-01執行start-dfs.sh

叢集關閉,在sht-sgmhadoopnn-01執行stop-dfs.sh

#####叢集啟動############

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 sbin]# start-dfs.sh
  2. 16/02/25 14:21:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  3. Starting namenodes on [sht-sgmhadoopnn-01 sht-sgmhadoopnn-02]
  4. sht-sgmhadoopnn-01: starting namenode, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-namenode-sht-sgmhadoopnn-01.telenav.cn.out
  5. sht-sgmhadoopnn-02: starting namenode, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-namenode-sht-sgmhadoopnn-02.telenav.cn.out
  6. sht-sgmhadoopdn-01: starting datanode, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-datanode-sht-sgmhadoopdn-01.telenav.cn.out
  7. sht-sgmhadoopdn-02: starting datanode, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-datanode-sht-sgmhadoopdn-02.telenav.cn.out
  8. sht-sgmhadoopdn-03: starting datanode, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-datanode-sht-sgmhadoopdn-03.telenav.cn.out
  9. Starting journal nodes [sht-sgmhadoopdn-01 sht-sgmhadoopdn-02 sht-sgmhadoopdn-03]
  10. sht-sgmhadoopdn-01: journalnode running as process 6348. Stop it first.
  11. sht-sgmhadoopdn-03: journalnode running as process 16722. Stop it first.
  12. sht-sgmhadoopdn-02: journalnode running as process 7197. Stop it first.
  13. 16/02/25 14:21:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  14. Starting ZK Failover Controllers on NN hosts [sht-sgmhadoopnn-01 sht-sgmhadoopnn-02]
  15. sht-sgmhadoopnn-01: starting zkfc, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-zkfc-sht-sgmhadoopnn-01.telenav.cn.out
  16. sht-sgmhadoopnn-02: starting zkfc, logging to /hadoop/hadoop-2.7.2/logs/hadoop-root-zkfc-sht-sgmhadoopnn-02.telenav.cn.out
  17. You have mail in /var/spool/mail/root

####單程式啟動###########

NameNode(sht-sgmhadoopnn-01, sht-sgmhadoopnn-02):

hadoop-daemon.sh start namenode

DataNode(sht-sgmhadoopdn-01, sht-sgmhadoopdn-02, sht-sgmhadoopdn-03):

hadoop-daemon.sh start datanode

JournamNode(sht-sgmhadoopdn-01, sht-sgmhadoopdn-02, sht-sgmhadoopdn-03):

hadoop-daemon.sh start journalnode

ZKFC(sht-sgmhadoopnn-01, sht-sgmhadoopnn-02):

hadoop-daemon.sh start zkfc

f.驗證namenode,datanode,zkfc

1) 程式

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 sbin]# jps
  2. 12712 Jps
  3. 12593 DFSZKFailoverController
  4. 12278 NameNode
  5. [root@sht-sgmhadoopnn-02 ~]# jps
  6. 29714 NameNode
  7. 29849 DFSZKFailoverController
  8. 30229 Jps
  9. [root@sht-sgmhadoopdn-01 ~]# jps
  10. 6348 JournalNode
  11. 8775 Jps
  12. 559 QuorumPeerMain
  13. 8509 DataNode
  14. [root@sht-sgmhadoopdn-02 ~]# jps
  15. 9430 Jps
  16. 9160 DataNode
  17. 7197 JournalNode
  18. 2073 QuorumPeerMain
  19. [root@sht-sgmhadoopdn-03 ~]# jps
  20. 16722 JournalNode
  21. 17369 Jps
  22. 15519 QuorumPeerMain
  23. 17214 DataNode

2) 頁面

sht-sgmhadoopnn-01:

sht-sgmhadoopnn-02:

g.啟動YARN運算框架

#####叢集啟動############

1) sht-sgmhadoopnn-01啟動Yarn，命令所在目錄：$HADOOP_HOME/sbin

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 sbin]# start-yarn.sh
  2. starting yarn daemons
  3. starting resourcemanager, logging to /hadoop/hadoop-2.7.2/logs/yarn-root-resourcemanager-sht-sgmhadoopnn-01.telenav.cn.out
  4. sht-sgmhadoopdn-03: starting nodemanager, logging to /hadoop/hadoop-2.7.2/logs/yarn-root-nodemanager-sht-sgmhadoopdn-03.telenav.cn.out
  5. sht-sgmhadoopdn-02: starting nodemanager, logging to /hadoop/hadoop-2.7.2/logs/yarn-root-nodemanager-sht-sgmhadoopdn-02.telenav.cn.out
  6. sht-sgmhadoopdn-01: starting nodemanager, logging to /hadoop/hadoop-2.7.2/logs/yarn-root-nodemanager-sht-sgmhadoopdn-01.telenav.cn.out

2) sht-sgmhadoopnn-02備機啟動RM

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-02 sbin]# yarn-daemon.sh start resourcemanager
  2. starting resourcemanager, logging to /hadoop/hadoop-2.7.2/logs/yarn-root-resourcemanager-sht-sgmhadoopnn-02.telenav.cn.out

####單程式啟動###########

1) ResourceManager(sht-sgmhadoopnn-01, sht-sgmhadoopnn-02)

yarn-daemon.sh start resourcemanager

2) NodeManager(sht-sgmhadoopdn-01, sht-sgmhadoopdn-02, sht-sgmhadoopdn-03)

yarn-daemon.sh start nodemanager

######關閉#############

[root@sht-sgmhadoopnn-01 sbin]# stop-yarn.sh

#包含namenode的resourcemanager程式，datanode的nodemanager程式

[root@sht-sgmhadoopnn-02 sbin]# yarn-daemon.sh stop resourcemanager

h.驗證resourcemanager,nodemanager

1) 程式

點選(此處)摺疊或開啟

  1. [root@sht-sgmhadoopnn-01 sbin]# jps
  2. 13611 Jps
  3. 12593 DFSZKFailoverController
  4. 12278 NameNode
  5. 13384 ResourceManager
  6. [root@sht-sgmhadoopnn-02 sbin]# jps
  7. 32265 ResourceManager
  8. 32304 Jps
  9. 29714 NameNode
  10. 29849 DFSZKFailoverController
  11. [root@sht-sgmhadoopdn-01 ~]# jps
  12. 6348 JournalNode
  13. 559 QuorumPeerMain
  14. 8509 DataNode
  15. 10286 NodeManager
  16. 10423 Jps
  17. [root@sht-sgmhadoopdn-02 ~]# jps
  18. 9160 DataNode
  19. 10909 NodeManager
  20. 11937 Jps
  21. 7197 JournalNode
  22. 2073 QuorumPeerMain
  23. [root@sht-sgmhadoopdn-03 ~]# jps
  24. 18031 Jps
  25. 16722 JournalNode
  26. 17710 NodeManager
  27. 15519 QuorumPeerMain
  28. 17214 DataNode

2) 頁面

ResourceManger（Active）：

ResourceManger（Standby）：

九.監控叢集

[root@sht-sgmhadoopnn-01 ~]# hdfs dfsadmin -report

十.附件及參考

#

#

hadoop : 

zookeeper :

參考:

Hadoop-2.3.0-cdh5.0.1完全分散式環境搭建(NameNode,ResourceManager HA):

http://blog.itpub.net/30089851/viewspace-1987620/

如何解決這類問題：The string "--" is not permitted within comments:

http://blog.csdn.net/free4294/article/details/38681095

SecureCRT連線linux終端中文顯示亂碼解決辦法:

http://www.cnblogs.com/qi09/archive/2013/02/05/2892922.html


參照:http://blog.itpub.net/30089851/viewspace-1987620/





來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/30089851/viewspace-1994585/，如需轉載，請註明出處，否則將追究法律責任。

相關文章