hadoop部署

Hadoop介紹:

廣義: 以apache hadoop軟體為主的生態圈(hive zookeeper spark hbase)

狹義: 單指apache hadoop軟體

相關官網：

hadoop.apache.org

hive.apache.org

spark.apache.org

cdh-hadoop：

hadoop軟體及版本:

1.x 企業不用

2.x 主流

3.x 沒有企業敢用

a.採坑

b.很多公司都是CDH5.x部署大資料環境 ()，即2.6.0-cdh5.7.0 =? apache hadoop2.6.0

很多公司都是CDH5.X部署大資料環境(),相當於是把一個生態圈的元件，集中成為一個系統。

作為基礎環境，裡面裝的2.6.0-cdh5.7.0，

注意此版本不等於apache hadoop2.6.0

，因為

cdh5.7.0中hadoop做了bug升級。

hadoop軟體:

hdfs：儲存 分散式檔案系統

mapreduce:計算。用java計算job1,job2,但企業不用java(開發難度大，程式碼複雜)

yarn: 資源和作業排程(cpu memory分配)，即：哪個作業分配到哪個節點中排程。

--如果需要按照ssh

Ubuntu Linux:

$ sudo apt-get install ssh

$ sudo apt-get install rsync

----------------------------------------------------------------------------------------------------

安裝部分：

環境：CentOS 偽分佈安裝：即

單節點安裝

HADOOP版本：hadoop-2.6.0-cdh5.7.0.tar.gz

JDK版本：jdk-8u45-linux-x64.gz

安裝原則：不同軟體需要指定對應的使用者

linux       root使用者

mysql     mysqladmin使用者

hadoop  hadoop使用者

1.建立hadoop使用者和上傳hadoop軟體

******************************

useradd hadoop

su - hadoop

mkdir app

cd app/

上傳hadoop包

結果如下：

[hadoop@hadoop app]$ pwd

/home/hadoop/app

[hadoop@hadoop app]$ ls -l

total 304288

drwxr-xr-x 15 hadoop hadoop      4096 Feb 14 23:37 hadoop-2.6.0-cdh5.7.0

-rw-r--r--  1 root   root   311585484 Feb 14 17:32 hadoop-2.6.0-cdh5.7.0.tar.gz

***********************************

2.部署jdk ,

要用CDH版本的JDK

***********************************

建立JDK目錄，上傳JDK包

要用CDH版本的JDK

su - root

mkdir /usr/java             #上傳JDK包到此目錄

mkdir /usr/share/java   #部署CDH環境時jdbc jar包需要放到此目錄，否則報錯

cd   /usr/java

tar   -xzvf     jdk-8u45-linux-x64.gz  #解壓JDK

drwxr-xr-x 8

uucp  143

     4096 Apr 11  2015 jdk1.8.0_45   #注意解壓後使用者、組是不對的，需要改使用者組為root:root

chown -R root:root jdk1.8.0_45

drwxr-xr-x 8

root root

     4096 Apr 11  2015 jdk1.8.0_45

結果如下：

[root@hadoop java]# pwd

/usr/java

[root@hadoop java]# ll

total 169216

drwxr-xr-x 8 root root      4096 Apr 11  2015 jdk1.8.0_45

-rw-r--r-- 1 root root 173271626 Jan 26 18:35 jdk-8u45-linux-x64.gz

*****************************************

3.設定java環境變數

su - root

vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_45

export JRE_HOME=$JAVA_HOME/jre

export CLASSPATH=.:$JAVA_HOME/lib:$JER_HOME/lib:$CLASSPATH

export PATH=$JAVA_HOME/bin:$JER_HOME/bin:$PATH

source /etc/profile

[root@hadoop java]#

which java

/usr/java/jdk1.8.0_45/bin/java

**********************

4.解壓hadoop

su - hadoop

cd  /home/hadoop/app

[hadoop@hadoop002 app]$

tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz

[hadoop@hadoop002 app]$

cd hadoop-2.6.0-cdh5.7.0

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ll

total 76

drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin  可執行指令碼

drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1

drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera

drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc  配置目錄(conf)

drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples

drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1

drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include

drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib  jar包目錄

drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec

drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 sbin hadoop元件的啟動 停止指令碼

drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share

drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

*********************************************************

4.解壓並配置hadoop

su - hadoop

cd app

tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz

cd  /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop

vi  core-site.xml

<configuration>

 
<property>

       <name>fs.defaultFS</name>

       <value>hdfs://localhost:9000</value>

   </property>

</configuration>

vi hdfs-site.xml

<configuration>

<property>

       <name>dfs.replication</name>

       <value>1</value>

   </property>

</configuration>

配置hadoop的環境變數，否則會在啟動時候報錯

vi /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/hadoop-env.sh

export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop

export JAVA_HOME=/usr/java/jdk1.8.0_45

*****************************

*****************************

5.配置ssh localhost無密碼信任關係

su - hadoop

ssh-keygen  #一直回車

cd .ssh   #可以看到兩個檔案

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  #生成authorized_keys信任檔案

ssh localhost date

The authenticity of host 'localhost (127.0.0.1)' can't be established.

RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.

Are you sure you want to continue connecting (yes/no)?

yes

Warning: Permanently added 'localhost' (RSA) to the list of known hosts.

Wed Feb 13 22:41:17 CST 2019

chmod 600 authorized_keys

  #

非常重要，如果不更改許可權，執行ssh localhost date時會讓輸入密碼，但hadoop使用者根本無密碼，此時就是許可權搞的貓膩。

**********************************

6.格式化

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

bin/hdfs namenode -format

***************************************

cd /home/hadoop/app/hadoop-2.6.0-cdh5.7.0

bin/hdfs namenode -format  #為何進入bin 再 hdfs namenode -format說找不到hdfs命令

***************************************

7.啟動hadoop服務

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

sbin/start-dfs.sh

19/02/13 22:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [localhost]

localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out

localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out

Starting secondary namenodes [0.0.0.0]

The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.

Are you sure you want to continue connecting (yes/no)?

yes

 #輸入yes,,因為

ssh 信任關係 是配置的是localhost，而非0.0.0.0

0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.

0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out

19/02/13 22:49:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

sbin/stop-dfs.sh

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

sbin/start-dfs.sh

19/02/13 22:57:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [localhost]

localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out

localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out

19/02/13 22:57:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

jps

 #檢驗是否正常啟動，需啟動以下四個服務

15059 Jps

14948 SecondaryNameNode 第二名稱節點 老二

14783 DataNode  資料節點  小弟

14655 NameNode  名稱節點  老大 讀寫

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

open  #安裝成功可以開啟hadoop的web管理介面:如圖

8.配置hadoop命令環境變數

***************************************************************

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

cat ~/.bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

export PATH=$HADOOP_PREFIX/bin:$PATH

source ~/.bash_profile

/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

***************************************************************

9.操作hadoop, hdfs dfs操作命令和Linux命令極其相似

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

bin/hdfs dfs -ls /

19/02/13 23:08:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

bin/hdfs dfs -ls /

19/02/13 23:11:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ls /

bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var

boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

bin/hdfs dfs -mkdir /ruozedata

19/02/13 23:11:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

bin/hdfs dfs -ls /

19/02/13 23:11:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Found 1 items

drwxr-xr-x   - hadoop supergroup          0 2019-02-13 23:11 /ruozedata

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ls /

bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var

boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

10.檢視幫助

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$  

bin/hdfs --help

作業:

1.ssh部落格 閱讀 摘抄

http://blog.itpub.net/30089851/viewspace-1992210/

http://blog.itpub.net/30089851/viewspace-2127102/

2.部署hdfs偽分散式

3.部落格要寫到hdfs偽分散式

小提示：

如果 su - zookeeper不能切換

解決方法：

更改：/etc/passwd中zookeeper使用者的登入方式由/sbin/nologin==>/bin/bash即可

hadoop實戰1--部署

相關文章