Ambari叢集搭建

OrangeCat_發表於2018-11-28

 

一.軟體要求

1.yum, rpm

2.scp curl unzip tar wget

3.jdk1.8.0

二.前提環境準備

2.1 ssh免密

生成master機器的ssh金鑰

# 生成master機器的公鑰和金鑰
[@hadoop.master01.com hadoop_install] ssh-keygen -t rsa

修改ssh登陸配置(禁用嚴格檢查主機) 

[@hadoop.master01.com hadoop_install] vim ~/.ssh/config
StrictHostKeyChecking no
UserKnownHostsFile /dev/null

將master機器的公鑰推送到其他機器上，${pass} ${ip}填寫你的主機ip和主機密碼，如果機器較多可以考慮寫個指令碼輪詢即可

[@hadoop.master01.com hadoop_install] sshpass -p ${pass} ssh-copy-id root@${ip}

 

2.2修改域名解析檔案以及自己的主機名

修改master機器上的/etc/hosts檔案

[@hadoop.master01.com hadoop_install] vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.134.65.173 hadoop.master01.com
10.143.46.213 hadoop.master02.com
10.139.53.168 hadoop.slave01.com
10.139.50.172 hadoop.slave02.com
10.142.103.174 hadoop.slave03.com

利用scp命令推送到其他機器上,機器比較多的時候利用指令碼輪詢 

[@hadoop.master01.com hadoop_install] scp /etc/hosts ${ip}:/etc/hosts

執行hostname命令（臨時修改）和/etc/hostname檔案 （永久修改），叢集裡面的其他機器也需要修改，機器較多的時候寫指令碼執行

[@hadoop.master01.com hadoop_install] hostname hadoop.master01.com
[@hadoop.master01.com hadoop_install] vim /etc/hostname
hadoop.master01.com

 

三.製作本地YUM源

3.1 安裝製作本地源工具

[@hadoop.master01.com hadoop_install] yum install yum-utils createrepo

3.2 安裝HTTP伺服器

[@hadoop.master01.com hadoop_install] yum install httpd -y

# 設定為開機自啟動並且啟動httpd服務
[@hadoop.master01.com hadoop_install] systemctl enable httpd && systemctl start httpd

3.3 下載系統對應的安裝包

由於叢集機器通常處於內網環境下，因此我們需要在外網機器上下載好對應的RPM包，最後通過lszrz命令上傳。如果沒有wget命令可以直接把URL複製到瀏覽器中，會自動執行下載

wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.2/ambari-2.6.2.2-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz
wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz

3.5 解壓安裝包

tar -zxvf ambari-2.6.2.2-centos7.tar.gz -C /var/www/html
tar -zxvf HDP-2.6.4.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz  -C /var/www/html/hdp/HDP-UTILS/

3.4 配置好對應的本地源（注意baseurl和gpgcheck裡面寫自己的repository的ip地址）

ambari.repo

#VERSION_NUMBER=2.6.2.2
[ambari-2.6.2.2]
name=ambari Version - ambari-2.6.2.2
baseurl=http://hadoop.master01.com/ambari/centos7/2.6.2.2-1
gpgcheck=1
gpgkey=http://hadoop.master01.com/ambari/centos7/2.6.2.2-1/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

 hdp.repo

#VERSION_NUMBER=2.6.4.0-91
[HDP-2.6.4.0]
name=HDP Version - HDP-2.6.4.0
baseurl=http://hadoop.master01.com/hdp/HDP/centos7/2.6.4.0-91
gpgcheck=1
gpgkey=http://hadoop.master01.com/hdp/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://hadoop.master01.com/hdp/HDP-UTILS
gpgcheck=1
gpgkey=http://hadoop.master01.com/hdp/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

四.安裝

4.1 Ambari-server(只在namenode節點上安裝，standbynamenode上不用)

yum install ambari-server -y

4.2 安裝java(直接在oracle java官網下載, 然後解壓到/usr/local/下)

關於java這個安裝操作請在叢集上的所有機器上執行,如果機器數量較多的話可以藉助scp功能推送

# 刪除自帶的java
[@hadoop.master01.com hadoop_install] yum remove java

# 刪除自帶的java開發包
[@hadoop.master01.com hadoop_install] yum remove java-devel

# 解壓我們自己的jdk
[@hadoop.master01.com hadoop_install] tar xvf jdk-8u191-linux-x64.tar.gz -C /usr/local

4.3 開始配置Ambari-server

[@hadoop.master01.com jdk1.8.0_191]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Ambari-server daemon is configured to run under user 'ambari'. Change this setting [y/n] (n)? y
Enter user account for ambari-server daemon (ambari):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk1.8.0_191       
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 1
Database admin user (postgres): 
Database name (ambari): 
Postgres schema (ambari): 
Username (ambari): 
Enter Database Password (123456): 
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Configuring local database...
Configuring PostgreSQL...
Backup for pg_hba found, reconfiguration not required
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

4.4 啟動ambari-server

請確保以下目錄的所屬使用者是ambari，否則可能啟動不了

/var/run/ambari-server/
/var/log/ambari-server/

啟動ambari-server

ambari-server start

啟動結束後可以在瀏覽器中輸入10.134.65.173:8080,檢查是否成功，這裡的10.134.65.173是我自己部署ambari-server的機器ip，如果你需要檢查的話替換以下ip便好.看到下面這個頁面，就說明啟動成功了

Username:admin

Password:admin

五.Hadoop服務安裝

 

 5.1 填寫叢集的名稱

5.2 選擇Hadoop整合的版本，並填寫repository地址

5.3 填寫叢集的主機域名和Master機器（部署ambari-server的那臺）私鑰

檢視私鑰

[@hadoop.master01.com jdk1.8.0_191] cat ~/.ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
MIIEpgIBAAKCAQEA5ansWVx8CKLj5tJGdu5rhd9+RJzvoWJFwTQ2+1eX8P0hrvsN
3hVJHF67tFqK4K5RQCWa9nQqJPDQO8Of3uo9gncAcfF4DHEA/dubYQc52iXBurC0
ZPtvxCH5L1lXt6FYQiIYphjpi3raighYCQUd+tfMF1yC8MXxkrHO+pfStLG2Xf4X
UgMfd1aGZjDrNNg+wL552AY3AzpzaoHTaTW08emqumgFwky3pMKV4xsM+8Biwme9
TL8cI6TlmXQXSixqGz2mQYMcK68czcNUe5WR0tp90oHt2sDa4QLoQmDvpqk9U/+D
Qp2QEjERUr4CLiUDKyhsEXZKyfn2A8X47E1dUwIDAQABAoIBAQDVnShdsb0YV1H9
FqY95uMRwN/0N7hXdk2FN+jP/RgxKXDUDWVHpKq8/Yw/Y1v7QYgHTzMEeuXcODSm
8fzYXS1bFQ7P7JYJlG4ksMekBxGTSz0XRVZ0+r1pQ2Q/2yzQevzZfldI3rETb/Cn
JL2lniHVeDS3oe+UvbhFBqeKw/5S25ldC9ROGw0OZgl52w1d+J/q38O+VhiZdnj6
Sb+vZX0QwG1zu5aV9k94iGl9DhyJRPKjOCmPc/mgLP6k8/mYTFIB8ntJH1LcQvF9
81ovp/zeQt4FwZf8cGnnwuwaWiTFiEd4iB8xQl0hBTpgW/ijmlJ6qwe1SOOA0zwp
qm/RWzuZAoGBAPzBOCNHhJiydcyuiLhBiG8dTuDd3GQTUUy31qZ1qeHaFSvir1fF
UZoowuPV+fgN7PFyzl1tSjOwxrxiizZF5iARFZ3Sw3TuxnUihjjytxvjn6VxwPKI
9ICf3q2xlYKuezWtF676ufnPRDaXgQFoRVjlVq5nkwaohzdp7k0/pGNHAoGBAOic
zljTQi9KwC2vBS/8QL9O1sBzGAjN2vAF9ThlForWKFZKNF4JW5nd8+LLhxXY3hSv
ria4lQkFUy1vdcJaimPVPynvZRqgPH6GaqK/YDyu8kpTRkOO4HJTiFuv9Z16JasW
a+Ot8F6BPRRLQas7PgtSvmh9Egjg6Jq9gxUxPEOVAoGBAKEgwBvldMcvRV+FiE0G
IebL5UP+y4fUtMhWjfUyH2mGcGa4rRUkCiBfidZay27x2bIWgS4YBOys7hScMakY
qFqa6Y/iE45QxZ1Ys/5L5KVX16dNvDYhTtVMlVVfVzOm7Y7LWMTIu5RgKFkw/iGa
ilLgaQDnsFfXSdPuqOcAAljzAoGBAOMW+shy7oFW+8ldvbW8EcXzBOr7MfKebei4
GE/HOLuwGc8sQHWxsxDDQ+8c/bNRdnZBkMZSm4UkArEvyIpGdCbEsBaTlWPqHUT0
mzRpm12C+h2fsNARs4NJk6zXWvUzo4Skrg2n6SuKBoRCeEjf3FGXpTZDLBEo6Uzw
SQGugIQFAoGBAPdIJIAzdJ9TyKKvQCOxqXoo5iYrQoqM6hy8MXYJeGWLbU6aszWb
yUO5smURZgFfm0LddP56Ubx7SQ5aBBVLgXaGTeYNFKuWWXqCim4hSx0+/r6W+6KJ
HozWYErDghVLrofED/GLHuQCFbLvkXQOvvKbtrGlfoX6Rn5/WK00SqjL
-----END RSA PRIVATE KEY-----

5.4 確認主機，檢查主機（這裡大家可能會失敗），

需要在/etc/ambari-agent/conf/ambari-agent.ini中增加一行配置（在叢集中的所有機器中操作）

[security]
force_https_protocol=PROTOCOL_TLSv1_2

增加完了別忘了重啟ambari-agent(注意不是ambari-server,在叢集中的所有機器中操作)

ambari-agent restart

看到下圖的時候就代表檢查主機成功！！！

 

後面的內容是關於hadoop叢集比較簡單的角色劃分部分了。。自己可以嘗試下，未完待續 

相關文章