Spark開發-HA環境的搭建

環境
1、Hadoop2.6
2、spark1.6
3、zookeeper 3.4.6

主機
192.168.18.140 master
192.168.18.141 slave1
192.168.18.142 slave2
192.168.18.143 slave3

現在我們要做的是，在master、slave1、slave2部署zookeeper。在master和slave1上面啟動spark的master，剛開始master為active的，slave1位standby的，當我們將master上面的spark程式kill以後，看是否可以切換到slave1上面來

安裝
1、zookeeper的安裝就不介紹了，詳細可以檢視文章 http://blog.csdn.net/paicmis/article/details/53264178

2、配置spark的環境資訊
修改spark-env.sh檔案

export JAVA_HOME=/opt/jdk1.7.0_79
export SCALA_HOME=/opt/scala-2.11.4
#export SPARK_MASTER_IP=192.168.18.140(主要是註釋這段，)
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=192.168.18.140:2181,192.168.18.141:2181,192.168.18.142:2181 -Dspark.deploy.zookeeper.dir=/spark"(這段的意思是告訴spark用zookeeper來協調，指定zookeeper的叢集和spark後設資料儲存的路徑)
export HADOOP_CONF_DIR=/opt/hadoop-2.6.5/etc/hadoop
export SPARK_WORKER_MEMORY=4g
export SPARK_EXECUTOR_MEMORY=1g
export SPARK_DRIVER_MEMORY=4G
export SPARK_WORKER_CORES=8

3、配置完成以後需要scp到其他的機器上面去

4、啟動 Hadoop zookeeper和spark
主要介紹一下spark的啟動過程
先在master上面執行 start-all.sh來啟動所有的程式，再在slave1上面啟動start-master.sh啟動master程式

[root@master sbin]# ./start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out
slave1: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.out
slave2: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.out
slave3: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave3.out

[root@slave1 sbin]# ./start-master.sh 
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-slave1.out
[root@slave1 sbin]# 

我們通過http://192.168.18.140:8080/的登入檢視是active

通過http://192.168.18.141:8080/檢視是standby的

驗證
通過下面這個命令我們啟動spark-shell來
[root@master bin]# ./spark-shell –master spark://192.168.18.140:7077,192.168.18.141:7077
通過日誌我們可以看到這個application向2個master地址傳送了請求

我們也可以發現在master上面有這個application進行

現在我們在master機器上面執行下面命令

[root@master sbin]# ./stop-master.sh 
stopping org.apache.spark.deploy.master.Master

我們可以看到application列印出瞭如下資訊了。master地址變化了

通過http://192.168.18.141:8080/檢視是active的，也可以看到application還是running的

我們通過http://192.168.18.140:8080/的登入檢視登入不了了

通過zookeeper我們也可以看到如下的資訊

[zk: localhost:2181(CONNECTED) 2] ls /spark
[master_status, leader_election]
[zk: localhost:2181(CONNECTED) 3] ls /spark/master_status
[app_app-20170920001354-0000, worker_worker-20170808072057-192.168.18.143-60628, worker_worker-20170920000720-192.168.18.142-45588, worker_worker-20170918011207-192.168.18.141-42902]
[zk: localhost:2181(CONNECTED) 4] ls /spark/leader_election
[_c_0e4c8c74-75de-4873-bb6a-13aa8dddc604-latch-0000000001]