Spark master standalong
~/.bashrc
export JAVA_HOME=/usr/java/latest/
export HADOOP_HOME=~/hadoop
export SPARK_HOME=~/spark
export PATH=.:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
export SPARK_MASTER_HOST=192.168.202.134
export SPARK_LOCAL_IP=192.168.202.134
spark/conf/spark-env.sh
export SPARK_LOCAL_IP=192.168.202.134
export SPARK_MASTER_HOST=192.168.202.134
Start spark master and slaves.
sh ./start-master.sh
sh ./start-slaves.sh
Submit spark task:
spark/bin/spark-submit --class SparkTest --master spark://192.168.202.134:7077 --deploy-mode cluster -
-supervise --total-executor-cores 2 andy.jar
Submit spark task on Windows, got the following error:
17/03/10 15:52:16 ERROR ClientEndpoint: Exception from cluster was: java.nio.file.NoSuchFileException: /D:/Tools/spark-2.1.0-bin-hadoop2.7/bin/andy.jar
java.nio.file.NoSuchFileException: /D:/Tools/spark-2.1.0-bin-hadoop2.7/bin/andy.jar
export JAVA_HOME=/usr/java/latest/
export HADOOP_HOME=~/hadoop
export SPARK_HOME=~/spark
export PATH=.:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
export SPARK_MASTER_HOST=192.168.202.134
export SPARK_LOCAL_IP=192.168.202.134
spark/conf/spark-env.sh
export SPARK_LOCAL_IP=192.168.202.134
export SPARK_MASTER_HOST=192.168.202.134
Start spark master and slaves.
sh ./start-master.sh
sh ./start-slaves.sh
Submit spark task:
spark/bin/spark-submit --class SparkTest --master spark://192.168.202.134:7077 --deploy-mode cluster -
-supervise --total-executor-cores 2 andy.jar
Submit spark task on Windows, got the following error:
17/03/10 15:52:16 ERROR ClientEndpoint: Exception from cluster was: java.nio.file.NoSuchFileException: /D:/Tools/spark-2.1.0-bin-hadoop2.7/bin/andy.jar
java.nio.file.NoSuchFileException: /D:/Tools/spark-2.1.0-bin-hadoop2.7/bin/andy.jar
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/10742815/viewspace-2135111/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- spark core原始碼分析3 Master HASpark原始碼AST
- 關於MASTER=spark://SparkMaster:7077 ./spark-shell問題ASTSpark
- 【Spark篇】---Spark中Master-HA和historyServer的搭建和應用SparkASTServer
- Spark Standalone模式 Master程式掛掉問題Spark模式AST
- spark core原始碼分析2 master啟動流程Spark原始碼AST
- MySQL Master/Slave Master/MasterMySqlAST
- ! [rejected] master -> master (fetch first)AST
- Mysql5.6 Master+MasterMySqlAST
- Mongodb的master-slave模式與master-master模式實驗MongoDBAST模式
- change master 未指定master_log_fileAST
- Git branching: master vs. origin/master vs. remotes/origin/masterGitASTREM
- show master logs 和 show master status 區別AST
- 【HBase】start master 與 start master --backup 的區別AST
- Spark之spark shellSpark
- 【Spark篇】---Spark初始Spark
- git rebase masterGitAST
- Redis master and slaveRedisAST
- MySQL master/slaveMySqlAST
- Scrum Master JobGPTScrumASTGPT
- Spark on Yarn 和Spark on MesosSparkYarn
- Spark系列 - (3) Spark SQLSparkSQL
- MySQL5.7 Master-Master主主搭建for Centos7MySqlASTCentOS
- 2 新增standby masterAST
- Master-Worker 模式AST模式
- Scrum Master 生存指南ScrumAST
- 1248:Dungeon MasterAST
- Master MVVM in Swift<1>ASTMVVMSwift
- Master Note for Tablespace IssuesAST
- Master-Worker模式AST模式
- 母版頁(Master Pages)AST
- MySql的Master/SlaveMySqlAST
- Master Data Management UpdateAST
- KSV master waitASTAI
- WPF master detail viewASTAIView
- DocumentDB 報錯“not master”AST
- C. Game MasterGAMAST
- master..spt_valuesAST
- Spark學習進度-Spark環境搭建&Spark shellSpark