Hadoop3偽分散式安裝指南

安裝環境：CentOS 7，Hadoop3.0

1.安裝JDK，設定環境變數

export JAVA_HOME=/opt/soft/jdk1.8.0_162
export PATH=$PATH:$JAVA_HOME/bin

2.解壓縮hadoop，設定環境變數

export HADOOP_HOME=/opt/soft/hadoop-3.0.0
export PATH=$PATH:$HADOOP_HOME/bin

3.編輯etc/hadoop/hadoop-env.sh，設定JAVA_HOME

export JAVA_HOME=/usr/java/latest

4.編輯core-site.xml

etc/hadoop/core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

5.編輯hdfs-site.xml

etc/hadoop/hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

6.設定ssh

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
$ ssh localhost

7.初始化hdfs的namenode

$ /opt/soft/hadoop-3.0.0/bin/hdfs namenode -format

8.編輯/opt/soft/hadoop-3.0.0/sbin/start-dfs.sh與/opt/soft/hadoop-3.0.0/sbin/stop-dfs.sh,在頂部新增

HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

9.啟動hdfs

$ /opt/soft/hadoop-3.0.0/sbin/start-dfs.sh
瀏覽器訪問NameNode - http://localhost:9870/

10.編輯mapred-site.xml

etc/hadoop/mapred-site.xml:

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

11.編輯yarn-site.xml

etc/hadoop/yarn-site.xml:

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

12.編輯/opt/soft/hadoop-3.0.0/sbin/start-yarn.sh與/opt/soft/hadoop-3.0.0/sbin/stop-yarn.sh,在頂部新增

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

13.開啟yarn

$ /opt/soft/hadoop-3.0.0/sbin/start-yarn.sh
瀏覽器訪問ResourceManager - http://localhost:8088/