大資料平臺搭建(1)

注意:因為部落格中美元符號有特殊含義,所以將美元符號替換為&
涉及技術:
JDk1.8+zookeeper-3.4.10+hadoop-2.7.3.tar.gz+
hbase-1.3.1+scala-2.11.8.tgz+spark-2.1.0-bin-hadoop2.7+
apache-hive-2.3.2-bin.tar.gz+
mysql-5.7.20-linux-glibc2.12-x86_64.tar.gz

1.機器準備(一共準備5臺機器,其中2臺作為namenode,3臺作為datanode)
D:\Virtual Machines\namenode01 namenode01 192.168.242.167 zookeeper
D:\Virtual Machines\namenode02 namenode02 192.168.242.168 zookeeper
D:\Virtual Machines\datanode01 datanode01 192.168.242.169 zookeeper
D:\Virtual Machines\datanode02 datanode02 192.168.242.170 zookeeper
D:\Virtual Machines\datanode03 datanode03 192.168.242.171 zookeeper

2.關於網路配置
a.主機名配置(vim /etc/sysconfig/network)
例如 HOSTNAME=namenode01
b.網路對映,即將IP與主機名繫結(vim /etc/hosts)
192.168.242.167 namenode01
192.168.242.168 namenode02
192.168.242.169 datanode01
192.168.242.170 datanode02
192.168.242.171 datanode03
c.禁用IPV6
echo ” “>>/etc/modprobe.d/dist.conf
echo “alias net-pf-10 off”>>/etc/modprobe.d/dist.conf
echo “alias ipv6 off”>>/etc/modprobe.d/dist.conf

3.重啟所有的機器
shutdown -r now

4.關閉防火牆
關閉防火牆 service iptables stop
防火牆開機不自動啟動 chkconfig iptables off
檢視防火牆狀況 chkconfig –list | grep iptables

5.關閉SELINUX
將SELINUX的值修改為disabled(vim /etc/syconfig/selinux)

6.ssh免密碼登陸
a.回車四次,/root/.ssh資料夾下生成id_rsa私鑰和id_rsa.pub公鑰(ssh-keygen)
b.拷貝公鑰到本機的公鑰檔案authorized_key(ssh-copy-id -i id_rsa.pub localhost)
c.拷貝公鑰檔案到其他機器的authorized_key中
(scp root@datanode03:/root/.ssh/authorized_keys /root/.ssh/)

7.系統時鐘同步(暫時略過)
a.檢視當前系統時間(date)
b.檢視當前系統時間伺服器的狀態(service ntpd status)
c.開啟當前系統的時間伺服器(service ntpd start)

8.安裝JDK
a.在指定目錄解壓JDK
b.配置JDK的環境變數(vim /etc/profile檔案)
c.使profile檔案生效(source /etc/profile)
9.安裝zookeeper
a.上傳:將zookeeper-3.4.10.tar.gz壓縮包上傳到/user/local路徑下
b.解壓:將上傳的zookeeper包解壓縮到當前的目錄下(tar -zxvf zookeeper-3.4.10.tar.gz)
c.配置環境變數:在profile檔案末尾新增配置(vim /etc/profile)
export ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.10
export PATH=&ZOOKEEPER_HOME/bin:&ZOOKEEPER_HOME/conf:&PATH
d.生效:是配置的profile檔案立刻生效(source /etc/profile)
e.配置zookeeper-3.4.10/conf/zoo.cfg檔案,這個檔案本身是沒有的,有個zoo_sample.cfg模板
進入conf目錄(cd zookeeper-3.4.5/conf)
拷貝模板(cp zoo_sample.cfg zoo.cfg)
修改zoo.cfg檔案,紅色是修改部分(vim zoo.cfg)
配置zoo.cfg內容如下:

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/usr/local/zookeeper-3.4.10/data

# the port at which the clients will connect

clientPort=2181

#

# Be sure to read the maintenance section of the # administrator guide before turning on autopurge.

#

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1

server.1=namenode01:2888:3888
server.2=namenode02:2888:3888
server.3=datanode01:2888:3888
server.4=datanode02:2888:3888
server.5=datanode03:2888:3888

f.建立/user/local/zookeeper-3.4.10/data目錄,並在data中建立myid檔案
進入指定目錄(cd /user/local/zookeeper-3.4.10)
建立目錄data(mkdir data)
進入data目錄(cd /user/local/zookeeper-3.4.10/data) 進入data資料夾下
建立myid檔案,並在這個文字內寫入1(touch myid)
g.測試zookeeper叢集是否搭建好
啟動zookeeper叢集:(zkServer.sh start)
檢視當前zookeeper叢集的狀態,一臺leader其他的為follower:(zkServer.sh status)
啟動客戶端:zkCli.sh -server namenode02:2181