Hadoop叢集部署安裝

一、規劃

使用VMWare Workstation建立3臺虛擬機器。定義如下表：

	Hostname	IP	OS
Master	n1	192.168.10.101	Cent OS 5
Slave1	d1	192.168.10.102	Cent OS 5
Slave2	d2	192.168.10.103	Cent OS 5

二、前期準備

1、在所有主機上，建立相同的使用者（組）

[root@n1 ~]# groupadd -g 500 hadoop

[root@n1 ~]# useradd -u 500 -g hadoop -G hadoop -d /home/hadoop hadoop

[root@n1 ~]# passwd hadoop

Changing password for user hadoop.

New UNIX password:

BAD PASSWORD: it is based on a dictionary word

Retype new UNIX password:

passwd: all authentication tokens updated successfully.

 

編輯/etc/hosts檔案，形如：

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1       localhost.localdomain localhost

::1             localhost6.localdomain6 localhost6

192.168.10.101  n1      n1

192.168.10.102  d1      d1

192.168.10.103  d2      d2

 

2、安裝JDK

使用WinSCP將JDK1.6安裝包傳送到伺服器主機上。

[root@n1 ~]# chown -R hadoop:hadoop /software

[root@n1 ~]# su - hadoop

[hadoop@n1 ~]$ ll /software/

total 70144

-rw-r--r-- 1 hadoop hadoop 71748681 Jul 15 22:33 jdk-6u33-linux-i586.bin

[hadoop@n1 ~]$ chmod -R 755 /software/

[hadoop@n1 ~]$ ll /software/

total 70144

-rwxr-xr-x 1 hadoop hadoop 71748681 Jul 15 22:33 jdk-6u33-linux-i586.bin

 

然後在另外兩個節點執行下面命令，使用scp複製jdk安裝包。

[root@d2 ~]# mkdir /software

[root@d2 ~]# scp root@192.168.10.101:/software/jdk-6u33-linux-i586.bin /software/.

The authenticity of host '192.168.10.101 (192.168.10.101)' can't be established.

RSA key fingerprint is 48:ef:fc:67:3f:43:88:0e:fa:e6:a6:ec:75:b0:c4:7f.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '192.168.10.101' (RSA) to the list of known hosts.

root@192.168.10.101's password:

jdk-6u33-linux-i586.bin                       100%   68MB   5.3MB/s   00:13

 

現在開始進行jdk安裝。

[root@n1 ~]# su - hadoop

[hadoop@n1 ~]$ cd /software/

[hadoop@n1 software]$ ll

total 70144

-rwxr-xr-x 1 hadoop hadoop 71748681 Oct  9 02:20 jdk-6u33-linux-i586.bin

[hadoop@n1 software]$ ./jdk-6u33-linux-i586.bin

Unpacking...

Checksumming... 

。。。。。。。 

Java(TM) SE Development Kit 6 successfully installed.

 

Product Registration is FREE and includes many benefits:

* Notification of new versions, patches, and updates

* Special offers on Oracle products, services and training

* Access to early releases and documentation

 

Product and system data will be collected. If your configuration

supports a browser, the JDK Product Registration form. will

be presented. If you do not register, none of this information

will be saved. You may also register your JDK later by

opening the register.html file (located in the JDK installation

directory) in a browser.

 

For more information on what data Registration collects and

how it is managed and used, see:

 

Press Enter to continue.....

 

 

Done.

 

配置JDK的環境變數。在.bash_profile檔案中加入如下程式碼。

JAVA_HOME=/software/jdk1.6.0_33

CLASSPATH=.:$JAVA_HOME/lib

PATH=$JAVA_HOME/bin:$PATH

 

export JAVA_HOME CLASSPATH PATH

別忘了執行

[hadoop@n1 ~]$ . .bash_profile

 

3、配置SSH互信

在n1上執行

[hadoop@n1 ~]$ mkdir .ssh

[hadoop@n1 ~]$ chmod 700 .ssh

[hadoop@n1 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

57:f2:fe:1b:e9:21:9f:fa:84:55:4c:13:37:d2:10:48 hadoop@n1

[hadoop@n1 ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_dsa.

Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.

The key fingerprint is:

24:6e:83:17:f0:5d:72:8f:00:9e:8b:7c:ed:28:9d:88 hadoop@n1

[hadoop@n1 ~]$ touch .ssh/authorized_keys

 

在d1上執行

[hadoop@d1 ~]$ mkdir .ssh

[hadoop@d1 ~]$ chmod 700 .ssh

[hadoop@d1 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

83:bc:0d:3b:fc:b8:53:e6:aa:cc:46:ed:2b:8d:8f:17 hadoop@d1

[hadoop@d1 ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_dsa.

Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.

The key fingerprint is:

86:7b:5e:66:99:22:7f:2c:38:07:08:73:91:70:3a:cb hadoop@d1

 

在d2上執行

[hadoop@d2 ~]$ mkdir .ssh

[hadoop@d2 ~]$ chmod 700 .ssh

[hadoop@d2 ~]$

[hadoop@d2 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

50:c1:c8:fa:8e:c0:0e:3e:eb:26:dc:23:ee:1d:27:fe hadoop@d2

[hadoop@d2 ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_dsa.

Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.

The key fingerprint is:

ae:f4:a2:96:2a:f2:24:33:c0:bc:63:08:2c:ef:af:51 hadoop@d2

 

然後在n1上繼續下面的執行。

[hadoop@n1 ~]$ cd .ssh

[hadoop@n1 .ssh]$ ll

total 16

-rw-rw-r-- 1 hadoop hadoop    0 Oct  9 02:49 authorized_keys

-rw------- 1 hadoop hadoop  668 Oct  9 02:48 id_dsa

-rw-r--r-- 1 hadoop hadoop  599 Oct  9 02:48 id_dsa.pub

-rw------- 1 hadoop hadoop 1675 Oct  9 02:48 id_rsa

-rw-r--r-- 1 hadoop hadoop  391 Oct  9 02:48 id_rsa.pub

[hadoop@n1 .ssh]$ ssh n1 cat /home/hadoop/.ssh/id_rsa.pub >> authorized_keys

The authenticity of host 'n1 (192.168.10.101)' can't be established.

RSA key fingerprint is 48:ef:fc:67:3f:43:88:0e:fa:e6:a6:ec:75:b0:c4:7f.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'n1,192.168.10.101' (RSA) to the list of known hosts.

hadoop@n1's password:

[hadoop@n1 .ssh]$ ssh n1 cat /home/hadoop/.ssh/id_dsa.pub >> authorized_keys

hadoop@n1's password:

[hadoop@n1 .ssh]$ ssh d1 cat /home/hadoop/.ssh/id_rsa.pub >> authorized_keys

The authenticity of host 'd1 (192.168.10.102)' can't be established.

RSA key fingerprint is 48:ef:fc:67:3f:43:88:0e:fa:e6:a6:ec:75:b0:c4:7f.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'd1,192.168.10.102' (RSA) to the list of known hosts.

hadoop@d1's password:

[hadoop@n1 .ssh]$ ssh d1 cat /home/hadoop/.ssh/id_dsa.pub >> authorized_keys

hadoop@d1's password:

[hadoop@n1 .ssh]$ ssh d2 cat /home/hadoop/.ssh/id_rsa.pub >> authorized_keys

The authenticity of host 'd2 (192.168.10.103)' can't be established.

RSA key fingerprint is 48:ef:fc:67:3f:43:88:0e:fa:e6:a6:ec:75:b0:c4:7f.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'd2,192.168.10.103' (RSA) to the list of known hosts.

hadoop@d2's password:

[hadoop@n1 .ssh]$ ssh d2 cat /home/hadoop/.ssh/id_dsa.pub >> authorized_keys

hadoop@d2's password:

[hadoop@n1 .ssh]$ ll

total 24

-rw-rw-r-- 1 hadoop hadoop 2970 Oct  9 02:56 authorized_keys

-rw------- 1 hadoop hadoop  668 Oct  9 02:48 id_dsa

-rw-r--r-- 1 hadoop hadoop  599 Oct  9 02:48 id_dsa.pub

-rw------- 1 hadoop hadoop 1675 Oct  9 02:48 id_rsa

-rw-r--r-- 1 hadoop hadoop  391 Oct  9 02:48 id_rsa.pub

-rw-r--r-- 1 hadoop hadoop 1197 Oct  9 02:56 known_hosts

 

然後，在3個節點上分別執行下面操作。

[hadoop@n1 ~]$ chmod 600 .ssh/authorized_keys

[hadoop@n1 ~]$ exec /ssh-agent $SHELL

-bash: /ssh-agent: No such file or directory

-bash: exec: /ssh-agent: cannot execute: No such file or directory

[hadoop@n1 ~]$ exec ssh-agent $SHELL

[hadoop@n1 ~]$ ssh-add

Identity added: /home/hadoop/.ssh/id_rsa (/home/hadoop/.ssh/id_rsa)

Identity added: /home/hadoop/.ssh/id_dsa (/home/hadoop/.ssh/id_dsa)

 

使用ssh 『機器名』date命令，驗證SSH互信的建立。

三、安裝hadoop

JDK和SSH已經安裝配置完成，可以開始部署Hadoop了。

再次使用WinSCP，將hadoop-0.20.2.tar.gz傳送到伺服器上；然後使用scp指令傳送到其它節點上。

[hadoop@n1 software]$ gunzip hadoop-0.20.2.tar.gz

[hadoop@n1 software]$ tar -xvf hadoop-0.20.2.tar

。。。。。

[hadoop@n1 software]$ mv hadoop-0.20.2 /home/hadoop/.

[hadoop@n1 ~]$ scp /software/hadoop-0.20.2.tar hadoop@192.168.10.102:/home/hadoop/.

hadoop-0.20.2.tar                             100%  129MB   6.2MB/s   00:21   

[hadoop@n1 ~]$ scp /software/hadoop-0.20.2.tar hadoop@192.168.10.103:/home/hadoop/.

hadoop-0.20.2.tar                             100%  129MB   5.6MB/s   00:23   

現在3個節點上的hadoop解壓完成，下面分別進行如下配置。

1）conf/Hadoop-env.sh檔案的修改，設定JAVA_HOME。

# The java implementation to use.  Required.

export JAVA_HOME=/software/jdk1.6.0_33

 

2）修改conf/core-site.xml

[hadoop@n1 conf]$ mkdir -p /software/hdata

        

                fs.default.name

                hdfs://n1:9000

       

 

3）修改conf/hdfs-site.xml

        

                dfs.data.dir

                /software/hdata

       

       

                dfs.replication

                2

       

 

4）修改conf/mapred-site.xml

        

                mapred.job.tracker

                n1:9001

       

 

5）修改conf/masters

n1

 

6）修改conf/slaves

d1

d2

 

7）將conf目錄及修改的檔案複製到其它兩個節點的相應目錄中

[hadoop@n1 conf]$ scp *.* d1:/home/hadoop/hadoop-0.20.2/conf/.

[hadoop@n1 conf]$ scp *.* d2:/home/hadoop/hadoop-0.20.2/conf/.

 

8）格式化hdfs

[hadoop@n1 hadoop-0.20.2]$ bin/hadoop namenode -format

12/10/09 04:12:40 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = n1/192.168.10.101

。。。。。。

12/10/09 04:12:41 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.

12/10/09 04:12:41 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at n1/192.168.10.101

************************************************************/

 

9）啟動hadoop

[hadoop@n1 hadoop-0.20.2]$ bin/start-all.sh

starting namenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-namenode-n1.out

d1: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-d1.out

d2: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-d2.out

n1: starting secondarynamenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-secondarynamenode-n1.out

starting jobtracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-jobtracker-n1.out

d2: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-d2.out

d1: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-d1.out

[hadoop@n1 hadoop-0.20.2]$

啟動成功，沒有報錯。

四、驗證

方法一：

[hadoop@n1 hadoop-0.20.2]$ bin/hadoop dfsadmin -report

Configured Capacity: 37176967168 (34.62 GB)

。。。。。。

DFS Remaining%: 75.57%

Last contact: Tue Oct 09 04:15:35 CST 2012

 

方法二：

[hadoop@n1 hadoop-0.20.2]$ /software/jdk1.6.0_33/bin/jps

30673 NameNode

。。。。。

[hadoop@n1 hadoop-0.20.2]$ ssh d1 /software/jdk1.6.0_33/bin/jps

30472 DataNode

。。。。。。 

[hadoop@n1 hadoop-0.20.2]$ ssh d2 /software/jdk1.6.0_33/bin/jps

30339 DataNode

。。。。。。

 

方法三：

使用瀏覽器，輸入n1:50070或n1:50030

自己使用瀏覽器看看吧，這裡就不貼圖啦！哈哈......