華為雲伺服器centos7.2下hadoop2.8.0完全分散式

前言:

使用雲伺服器有一個特別需要注意的地方,就是 /etc/hosts 檔案中
ip地址要是私網ip,不能用公網ip,否則報 java.net.BindException 

2017-06-28 15:25:07,371 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.net.BindException: Problem binding to [ecs-node-0001:50070] java.net.BindException: Cannot assign requested address; For more details see:  http://wiki.apache.org/hadoop/BindException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
        at org.apache.hadoop.ipc.Server.bind(Server.java:552)
        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:919)
        at org.apache.hadoop.ipc.Server.<init>(Server.java:2587)
        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:958)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:366)
        at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:341)
        at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:800)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:415)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:755)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:697)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:898)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:877)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1603)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1671)
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.apache.hadoop.ipc.Server.bind(Server.java:535)
        ... 13 more

這個錯誤折磨了我三天,剛開始完全不知道怎麼回事,
總是以為自己的配置檔案寫錯了,到處找配置檔案的例項,後來在 stackoverflow.com 找到了問題的原因
Problem was with azure server, it did not allow to bind on PUBLIC VIRTUAL IP (VIP) ADDRESS,
 it just allow binding on INTERNAL IP ADDRESS, so I create [virtual network]
(https://en.wikipedia.org/wiki/Virtual_network), 
add all 3 nodes to the virtual network.
https://stackoverflow.com/questions/35313781/failed-to-start-namenode-cannot-assign-requested-address

根據這個提示,測試了華為雲伺服器和阿里雲伺服器,真的好使.成功啟動了!
解決了這個問題之後,啟動hadoop機群就非常簡單了!

1.主機資訊

6臺華為雲伺服器
注意一定是
私有IP地址
192.168.1.197  ecs-node-0001   namenode yarn  節點
192.168.1.201  ecs-node-0002
192.168.1.200  ecs-node-0003
192.168.1.199  ecs-node-0004
192.168.1.198  ecs-node-0005
192.168.1.202  ecs-node-0006

2.免密登陸設定

可以參看我的上一篇文章,有詳細說明
http://www.jianshu.com/p/0cc626fcf4f5

以下操作,6臺都要設定
[root@ecs-4cbe-0001 ~]# vim /etc/hosts
新增
192.168.1.197  ecs-node-0001
192.168.1.201  ecs-node-0002
192.168.1.200  ecs-node-0003
192.168.1.199  ecs-node-0004
192.168.1.198  ecs-node-0005
192.168.1.202  ecs-node-0006
[root@ecs-4cbe-0001 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
59:a8:ba:10:e5:88:e2:1c:c0:7c:ec:3f:91:1a:fd:79 root@ecs-4cbe-0001
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|o .      .       |
|.o o.   . .      |
|..o+. .. o       |
|o.oo.+. S        |
|+ ..+.o .        |
| o...o o E       |
|   . .. .        |
|    .            |
+-----------------+
[root@ecs-4cbe-0001 ~]# 

以下是 ecs-node-0001上的操作:

[root@ecs-4cbe-0001 ~]# sh getId_rsa.pub.sh
[root@ecs-4cbe-0001 ~]# sh sendAuthorized_keys.sh 
[root@ecs-4cbe-0001 ~]# cat getId_rsa.pub.sh 
#!/bin/bash
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
for i in $( seq 2 6 )
do
   scp root@ecs-node-000${i}:~/.ssh/id_rsa.pub  ~/.ssh/p${i} 
   cat ~/.ssh/p${i} >> ~/.ssh/authorized_keys
done
[root@ecs-4cbe-0001 ~]# cat sendAuthorized_keys.sh 
#!/bin/bash
for i in $( seq 2 6 )
do
   scp ~/.ssh/authorized_keys  root@ecs-node-000${i}:~/.ssh
done
[root@ecs-4cbe-0001 ~]# 

免密登陸配置成功

3.更改hadoop2.8.0配置檔案

etc/hadoop/core-site.xml
<configuration>
<property>
   <name>fs.defaultFS</name>
   <value>hdfs://ecs-node-0001:9000/</value>
</property>
</configuration>

etc/hadoop/yarn-site.xml
<configuration>
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>ecs-node-0001</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
</configuration>

etc/hadoop/hdfs-site.xml
<configuration>
<property>
      <name>dfs.namenode.secondary.http-address</name>
      <value>ecs-node-0001:9001</value>
</property>
<property>
   <name>dfs.replication</name>
   <value>3</value>
</property>
</configuration>

etc/hadoop/mapred-site.xml
此檔案沒有,自己複製一個
[root@ecs-4cbe-0001 hadoop]# cat mapred-site.xml.template > mapred-site.xml

<configuration>
<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>
</configuration>

etc/hadoop/slaves檔案
ecs-node-0002
ecs-node-0003
ecs-node-0004
ecs-node-0005
ecs-node-0006

最後,把ecs-node-0001節點上的etc/hadoop 檔案分發到其它5個節點

[root@ecs-4cbe-0001 hadoop]# cd ~
[root@ecs-4cbe-0001 ~]# sh   sendHadoop.sh 
[root@ecs-4cbe-0001 ~]# cat sendHadoop.sh 
#!/bin/bash
for i in $( seq 2 6 )
do
   scp -r /usr/local/src/hadoop/hadoop/etc/hadoop  root@ecs-node-000${i}:/usr/local/src/hadoop/hadoop/etc
done

[root@ecs-4cbe-0001 ~]# 

4.啟動hadoop機群

[root@ecs-4cbe-0001 ~]# source goToHadoopRootDir.sh 
[root@ecs-4cbe-0001 hadoop]# bin/hdfs namenode -format
[root@ecs-4cbe-0001 hadoop]# sbin/start-dfs.sh
Starting namenodes on [ecs-node-0001]
ecs-node-0001: Warning: Permanently added 'ecs-node-0001,192.168.1.197' (ECDSA) to the list of known hosts.
ecs-node-0001: starting namenode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-namenode-ecs-4cbe-0001.out
ecs-node-0006: Warning: Permanently added 'ecs-node-0006,192.168.1.202' (ECDSA) to the list of known hosts.
ecs-node-0005: Warning: Permanently added 'ecs-node-0005,192.168.1.198' (ECDSA) to the list of known hosts.
ecs-node-0002: Warning: Permanently added 'ecs-node-0002,192.168.1.201' (ECDSA) to the list of known hosts.
ecs-node-0004: Warning: Permanently added 'ecs-node-0004,192.168.1.199' (ECDSA) to the list of known hosts.
ecs-node-0003: Warning: Permanently added 'ecs-node-0003,192.168.1.200' (ECDSA) to the list of known hosts.
ecs-node-0006: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0006.out
ecs-node-0005: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0005.out
ecs-node-0003: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0003.out
ecs-node-0002: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0002.out
ecs-node-0004: starting datanode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-datanode-ecs-4cbe-0004.out
Starting secondary namenodes [ecs-node-0001]
ecs-node-0001: Warning: Permanently added 'ecs-node-0001,192.168.1.197' (ECDSA) to the list of known hosts.
ecs-node-0001: starting secondarynamenode, logging to /usr/local/src/hadoop/hadoop/logs/hadoop-root-secondarynamenode-ecs-4cbe-0001.out
[root@ecs-4cbe-0001 hadoop]# jps
5936 Jps
5810 SecondaryNameNode
5609 NameNode
[root@ecs-4cbe-0001 hadoop]# 
[root@ecs-4cbe-0001 hadoop]# sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-resourcemanager-ecs-4cbe-0001.out
ecs-node-0006: Warning: Permanently added 'ecs-node-0006,192.168.1.202' (ECDSA) to the list of known hosts.
ecs-node-0004: Warning: Permanently added 'ecs-node-0004,192.168.1.199' (ECDSA) to the list of known hosts.
ecs-node-0002: Warning: Permanently added 'ecs-node-0002,192.168.1.201' (ECDSA) to the list of known hosts.
ecs-node-0003: Warning: Permanently added 'ecs-node-0003,192.168.1.200' (ECDSA) to the list of known hosts.
ecs-node-0005: Warning: Permanently added 'ecs-node-0005,192.168.1.198' (ECDSA) to the list of known hosts.
ecs-node-0002: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0002.out
ecs-node-0004: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0004.out
ecs-node-0006: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0006.out
ecs-node-0003: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0003.out
ecs-node-0005: starting nodemanager, logging to /usr/local/src/hadoop/hadoop/logs/yarn-root-nodemanager-ecs-4cbe-0005.out
[root@ecs-4cbe-0001 hadoop]# jps
5810 SecondaryNameNode
6259 Jps
5609 NameNode
5997 ResourceManager
[root@ecs-4cbe-0001 hadoop]# 


[root@ecs-4cbe-0004 ~]# jps
5714 Jps
5587 NodeManager
5447 DataNode
[root@ecs-4cbe-0004 ~]# 


hadoop機群成功啟動

5.關閉hadoop機群

[root@ecs-4cbe-0001 hadoop]#   sbin/stop-yarn.sh
[root@ecs-4cbe-0001 hadoop]#   sbin/stop-dfs.sh

大功告成!
忙了三天終於成功了,自己探索獲的果實是最美好的,也是最牢固的!