hadoop實戰1--部署

shaozi74108發表於2019-02-17
Hadoop
hadoop部署
Hadoop介紹:
廣義: 以apache hadoop軟體為主的生態圈(hive zookeeper spark hbase)
狹義: 單指apache hadoop軟體
相關官網：
hadoop.apache.org
hive.apache.org
spark.apache.org
cdh-hadoop：
hadoop軟體及版本:
1.x 企業不用
2.x 主流
3.x 沒有企業敢用
a.採坑
b.很多公司都是CDH5.x部署大資料環境 ()，即2.6.0-cdh5.7.0 =? apache hadoop2.6.0
很多公司都是CDH5.X部署大資料環境(),相當於是把一個生態圈的元件，集中成為一個系統。
作為基礎環境，裡面裝的2.6.0-cdh5.7.0， 注意此版本不等於apache hadoop2.6.0 ，因為
cdh5.7.0中hadoop做了bug升級。
hadoop軟體:
hdfs：儲存 分散式檔案系統
mapreduce:計算。用java計算job1,job2,但企業不用java(開發難度大，程式碼複雜)
yarn: 資源和作業排程(cpu memory分配)，即：哪個作業分配到哪個節點中排程。
--如果需要按照ssh
Ubuntu Linux:
$ sudo apt-get install ssh
$ sudo apt-get install rsync
----------------------------------------------------------------------------------------------------
安裝部分：
環境：CentOS 偽分佈安裝：即 單節點安裝
HADOOP版本：hadoop-2.6.0-cdh5.7.0.tar.gz
JDK版本：jdk-8u45-linux-x64.gz
安裝原則：不同軟體需要指定對應的使用者
linux       root使用者
mysql     mysqladmin使用者
hadoop  hadoop使用者
1.建立hadoop使用者和上傳hadoop軟體
******************************
useradd hadoop
su - hadoop
mkdir app
cd app/
上傳hadoop包
結果如下：
[hadoop@hadoop app]$ pwd
/home/hadoop/app
[hadoop@hadoop app]$ ls -l
total 304288
drwxr-xr-x 15 hadoop hadoop      4096 Feb 14 23:37 hadoop-2.6.0-cdh5.7.0
-rw-r--r--  1 root   root   311585484 Feb 14 17:32 hadoop-2.6.0-cdh5.7.0.tar.gz
***********************************
2.部署jdk , 要用CDH版本的JDK
***********************************
建立JDK目錄，上傳JDK包 要用CDH版本的JDK
su - root
mkdir /usr/java             #上傳JDK包到此目錄
mkdir /usr/share/java   #部署CDH環境時jdbc jar包需要放到此目錄，否則報錯
cd   /usr/java
tar   -xzvf     jdk-8u45-linux-x64.gz  #解壓JDK
drwxr-xr-x 8 uucp  143      4096 Apr 11  2015 jdk1.8.0_45   #注意解壓後使用者、組是不對的，需要改使用者組為root:root
chown -R root:root jdk1.8.0_45
drwxr-xr-x 8 root root      4096 Apr 11  2015 jdk1.8.0_45
結果如下：
[root@hadoop java]# pwd
/usr/java
[root@hadoop java]# ll
total 169216
drwxr-xr-x 8 root root      4096 Apr 11  2015 jdk1.8.0_45
-rw-r--r-- 1 root root 173271626 Jan 26 18:35 jdk-8u45-linux-x64.gz
*****************************************
3.設定java環境變數
su - root
vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JER_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JER_HOME/bin:$PATH
source /etc/profile
[root@hadoop java]# which java
/usr/java/jdk1.8.0_45/bin/java
**********************
4.解壓hadoop
su - hadoop
cd  /home/hadoop/app
[hadoop@hadoop002 app]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@hadoop002 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ll
total 76
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin  可執行指令碼
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera
drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc  配置目錄(conf)
drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib  jar包目錄
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 sbin hadoop元件的啟動 停止指令碼
drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share
drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
*********************************************************
4.解壓並配置hadoop
su - hadoop
cd app
tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz


cd  /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
vi  core-site.xml
<configuration>
  <property>
       <name>fs.defaultFS</name>
       <value>hdfs://localhost:9000</value>
   </property>
</configuration>
vi hdfs-site.xml
<configuration>
<property>
       <name>dfs.replication</name>
       <value>1</value>
   </property>
</configuration>
配置hadoop的環境變數，否則會在啟動時候報錯
vi /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/hadoop-env.sh
export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
export JAVA_HOME=/usr/java/jdk1.8.0_45
*****************************
*****************************
5.配置ssh localhost無密碼信任關係
su - hadoop
ssh-keygen  #一直回車
cd .ssh   #可以看到兩個檔案


cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  #生成authorized_keys信任檔案
ssh localhost date
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Wed Feb 13 22:41:17 CST 2019
chmod 600 authorized_keys   # 非常重要，如果不更改許可權，執行ssh localhost date時會讓輸入密碼，但hadoop使用者根本無密碼，此時就是許可權搞的貓膩。
**********************************
6.格式化
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format
***************************************
cd /home/hadoop/app/hadoop-2.6.0-cdh5.7.0
bin/hdfs namenode -format  #為何進入bin 再 hdfs namenode -format說找不到hdfs命令
***************************************
7.啟動hadoop服務
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
19/02/13 22:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out
localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
Are you sure you want to continue connecting (yes/no)? yes  #輸入yes,,因為 ssh 信任關係 是配置的是localhost，而非0.0.0.0
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out
19/02/13 22:49:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/stop-dfs.sh
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
19/02/13 22:57:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out
localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out
19/02/13 22:57:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ jps  #檢驗是否正常啟動，需啟動以下四個服務
15059 Jps
14948 SecondaryNameNode 第二名稱節點 老二
14783 DataNode  資料節點  小弟
14655 NameNode  名稱節點  老大 讀寫
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
open  #安裝成功可以開啟hadoop的web管理介面:如圖


8.配置hadoop命令環境變數
***************************************************************
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ cat ~/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_PREFIX/bin:$PATH
source ~/.bash_profile
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
***************************************************************
9.操作hadoop, hdfs dfs操作命令和Linux命令極其相似
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -ls /
19/02/13 23:08:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -ls /
19/02/13 23:11:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ls /
bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var
boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -mkdir /ruozedata
19/02/13 23:11:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -ls /
19/02/13 23:11:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-02-13 23:11 /ruozedata
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ls /
bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var
boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$
10.檢視幫助
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$   bin/hdfs --help
作業:
1.ssh部落格 閱讀 摘抄
2.部署hdfs偽分散式
3.部落格要寫到hdfs偽分散式
小提示：
如果 su - zookeeper不能切換
解決方法：
更改：/etc/passwd中zookeeper使用者的登入方式由/sbin/nologin==>/bin/bash即可


來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/28339956/viewspace-2636207/，如需轉載，請註明出處，否則將追究法律責任。

相關文章