Spark修煉之道（高階篇）——Spark原始碼閱讀：第一節 Spark應用程式提交流程
spark-submit 指令碼應用程式提交流程
在執行Spar應用程式時，會將spark應用程式打包後使用spark-submit指令碼提交到Spark中執行，執行提交命令如下：
<code class="hljs d has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">root<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">@sparkmaster</span>:/hadoopLearning/spark-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span>.0-bin-hadoop2.4/bin# ./spark-submit --master spark:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//sparkmaster:7077 </span> --<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> SparkWordCount --executor-memory <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>g /root/IdeaProjects/SparkWordCount/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">out</span>/artifacts/SparkWord Count_jar/SparkWordCount.jar hdfs:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//ns1/README.md</span> hdfs:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//ns1/SparkWordCountResult</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>
為弄清楚整個流程，我們先來分析一下spark-submit指令碼，spark-submit指令碼內容如下：
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-shebang" style="color: rgb(0, 102, 102); box-sizing: border-box;">#!/usr/bin/env bash </span> SPARK_HOME=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(cd "`dirname "$0"`"/..; pwd)</span>"</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># disable randomized hash for string in Python 3.3+</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PYTHONHASHSEED=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#spark-submit最終呼叫的是spark-class指令碼</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#傳入的類是org.apache.spark.deploy.SparkSubmit</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#及其它傳入的引數，如deploy mode、executor-memory等</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exec</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>"</span>/bin/spark-class org.apache.spark.deploy.SparkSubmit <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$@</span>"</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li></ul>
spark-class指令碼會載入spark配置的環境變數資訊、定位依賴包spark-assembly-1.5.0-hadoop2.4.0.jar檔案（以spark1.5.0為例）等，然後再呼叫org.apache.spark.launcher.Main正式啟動Spark應用程式的執行，具體如下：
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Figure out where Spark is installed</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#定位SAPRK_HOME目錄</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> SPARK_HOME=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(cd "`dirname "$0"`"/..; pwd)</span>"</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#載入load-spark-env.sh，執行環境相關資訊</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#例如配置檔案conf下的spark-env.sh等</span>
. <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>"</span>/bin/load-spark-env.sh
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Find the java binary</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># 定位JAVA_HOME目錄</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ -n <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${JAVA_HOME}</span>"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
RUNNER=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${JAVA_HOME}</span>/bin/java"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ `command -v java` ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
RUNNER=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"java"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span>
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"JAVA_HOME is not set"</span> >&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exit</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Find assembly jar</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#定位spark-assembly-1.5.0-hadoop2.4.0.jar檔案（以spark1.5.0為例）</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#這意味著任務提交時無需將該JAR檔案打包</span>
SPARK_ASSEMBLY_JAR=
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ <span class="hljs-operator" style="box-sizing: border-box;">-f</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/RELEASE"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
ASSEMBLY_DIR=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/lib"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span>
ASSEMBLY_DIR=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/assembly/target/scala-<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_SCALA_VERSION</span>"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>
num_jars=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(ls -1 "$ASSEMBLY_DIR" | grep "^spark-assembly.*hadoop.*\.jar$" | wc -l)</span>"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$num_jars</span>"</span> <span class="hljs-operator" style="box-sizing: border-box;">-eq</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"0"</span> <span class="hljs-operator" style="box-sizing: border-box;">-a</span> -z <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_ASSEMBLY_JAR</span>"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Failed to find Spark assembly in <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ASSEMBLY_DIR</span>."</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"You need to build Spark before running this program."</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exit</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>
ASSEMBLY_JARS=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(ls -1 "$ASSEMBLY_DIR" | grep "^spark-assembly.*hadoop.*\.jar$" || true)</span>"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$num_jars</span>"</span> <span class="hljs-operator" style="box-sizing: border-box;">-gt</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"1"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Found multiple Spark assembly jars in <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ASSEMBLY_DIR</span>:"</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ASSEMBLY_JARS</span>"</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Please remove all but one jar."</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exit</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>
SPARK_ASSEMBLY_JAR=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${ASSEMBLY_DIR}</span>/<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${ASSEMBLY_JARS}</span>"</span>
LAUNCH_CLASSPATH=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_ASSEMBLY_JAR</span>"</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Add the launcher build dir to the classpath if requested.</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ -n <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_PREPEND_CLASSES</span>"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
LAUNCH_CLASSPATH=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/launcher/target/scala-<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_SCALA_VERSION</span>/classes:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$LAUNCH_CLASSPATH</span>"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> _SPARK_ASSEMBLY=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_ASSEMBLY_JAR</span>"</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># The launcher library will print arguments separated by a NULL character, to allow arguments with</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># characters that would be otherwise interpreted by the shell. Read that in a while loop, populating</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># an array that will be used to exec the final command.</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#執行org.apache.spark.launcher.Main作為Spark應用程式的主入口</span>
CMD=()
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">while</span> IFS= <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">read</span> <span class="hljs-operator" style="box-sizing: border-box;">-d</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">''</span> -r ARG; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">do</span>
CMD+=(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ARG</span>"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">done</span> < <(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$RUNNER</span>"</span> -cp <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$LAUNCH_CLASSPATH</span>"</span> org.apache.spark.launcher.Main <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$@</span>"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exec</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${CMD[@]}</span>"</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li></ul>
從上述程式碼中，可以看到，通過org.apache.spark.launcher.Main類啟動org.apache.spark.deploy.SparkSubmit的執行，SparkSubmit部分原始碼如下：
<code class="hljs php has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//SparkSubmit Main方法</span>
def main(args: <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">Array</span>[String]): Unit = {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//任務提交時設定的引數，見圖2</span>
val appArgs = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkSubmitAarguments(args)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (appArgs.verbose) {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
printStream.println(appArgs)
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>
}
appArgs.action match {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//任務提交時，執行submit(appArgs)</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> SparkSubmitAction.SUBMIT => submit(appArgs)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> SparkSubmitAction.KILL => kill(appArgs)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)
}
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li></ul>
圖1 appArgs = new SparkSubmitAarguments(args)引數
進入submit方法：
<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"> <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
* Submit the application using the provided parameters.
*
* This runs in two steps. First, we prepare the launch environment by setting up
* the appropriate classpath, system properties, and application arguments for
* running the child main class based on the cluster manager and the deploy mode.
* Second, we use this launch environment to invoke the main method of the child
* main class.
*/</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> submit(args: SparkSubmitArguments): Unit = {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//執行引數等資訊，見圖2</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> (childArgs, childClasspath, sysProps, childMainClass) = prepareSubmitEnvironment(args)
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//定義在submit方法中的方法doRunMain()</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> doRunMain(): Unit = {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (args.proxyUser != <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>) {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> proxyUser = UserGroupInformation.createProxyUser(args.proxyUser,
UserGroupInformation.getCurrentUser())
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
proxyUser.doAs(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> PrivilegedExceptionAction[Unit]() {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">override</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> run(): Unit = {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//執行runMain方法</span>
runMain(childArgs, childClasspath, sysProps, childMainClass, args.verbose)
}
})
} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: Exception =>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Hadoop's AuthorizationException suppresses the exception's stack trace, which</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// makes the message printed to the output by the JVM not very helpful. Instead,</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// detect exceptions with empty stack traces here, and treat them differently.</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (e.getStackTrace().length == <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"ERROR: ${e.getClass().getName()}: ${e.getMessage()}"</span>)
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>
exitFn(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)
} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> e
}
}
} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//執行runMain方法</span>
runMain(childArgs, childClasspath, sysProps, childMainClass, args.verbose)
}
}
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// In standalone cluster mode, there are two submission gateways:</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// (1) The traditional Akka gateway using o.a.s.deploy.Client as a wrapper</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// (2) The new REST-based gateway introduced in Spark 1.3</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// The latter is the default behavior as of Spark 1.3, but Spark submit will fail over</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// to use the legacy gateway if the master endpoint turns out to be not a REST server.</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (args.isStandaloneCluster && args.useRest) {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
printStream.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Running Spark using the REST application submission protocol."</span>)
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//呼叫submit方法中的doRunMain方法</span>
doRunMain()
} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Fail over to use the legacy submission gateway</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: SubmitRestConnectionException =>
printWarning(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Master endpoint ${args.master} was not a REST server. "</span> +
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Falling back to legacy submission gateway instead."</span>)
args.useRest = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>
submit(args)
}
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// In all other modes, just run the main class as prepared</span>
} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//呼叫submit方法中的doRunMain方法</span>
doRunMain()
}
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li></ul>
圖2 任務提交時設定的引數，
從上面的程式碼可以看到，最終呼叫的是runMain方法，其原始碼如下：
<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
* Run the main method of the child class using the provided launch environment.
*
* Note that this main class will not be the one provided by the user if we're
* running cluster deploy mode or python applications.
*/</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> runMain(
childArgs: Seq[String],
childClasspath: Seq[String],
sysProps: Map[String, String],
childMainClass: String,
verbose: Boolean): Unit = {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (verbose) {
printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Main class:\n$childMainClass"</span>)
printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Arguments:\n${childArgs.mkString("</span>\n<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")}"</span>)
printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"System properties:\n${sysProps.mkString("</span>\n<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")}"</span>)
printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Classpath elements:\n${childClasspath.mkString("</span>\n<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")}"</span>)
printStream.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\n"</span>)
}
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> loader =
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (sysProps.getOrElse(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark.driver.userClassPathFirst"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"false"</span>).toBoolean) {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> ChildFirstURLClassLoader(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Array[URL](<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),
Thread.currentThread.getContextClassLoader)
} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> MutableURLClassLoader(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Array[URL](<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),
Thread.currentThread.getContextClassLoader)
}
Thread.currentThread.setContextClassLoader(loader)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (jar <- childClasspath) {
addJarToClasspath(jar, loader)
}
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> ((key, value) <- sysProps) {
System.setProperty(key, value)
}
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">var</span> mainClass: Class[_] = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//複用反射載入childMainClass，這裡為SparkWordCount</span>
mainClass = Utils.classForName(childMainClass)
} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: ClassNotFoundException =>
e.printStackTrace(printStream)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (childMainClass.contains(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"thriftserver"</span>)) {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Failed to load main class $childMainClass."</span>)
printStream.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"You need to build Spark with -Phive and -Phive-thriftserver."</span>)
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>
}
System.exit(CLASS_NOT_FOUND_EXIT_STATUS)
}
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// SPARK-4170</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (classOf[scala.App].isAssignableFrom(mainClass)) {
printWarning(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Subclasses of scala.App may not work correctly. Use a main() method instead."</span>)
}
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//呼叫反射機制載入main方法,即SparkWordCount中的main方法</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> mainMethod = mainClass.getMethod(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"main"</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Array[String](<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>).getClass)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (!Modifier.isStatic(mainMethod.getModifiers)) {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> IllegalStateException(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"The main method in the given main class must be static"</span>)
}
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> findCause(t: Throwable): Throwable = t <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">match</span> {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: UndeclaredThrowableException =>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (e.getCause() != <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>) findCause(e.getCause()) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> e
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: InvocationTargetException =>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (e.getCause() != <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>) findCause(e.getCause()) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> e
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: Throwable =>
e
}
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//執行main方法，即執行SparkWordCount</span>
mainMethod.invoke(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>, childArgs.toArray)
} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> t: Throwable =>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> findCause(t)
}
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li></ul>
mainMethod.invoke(null, childArgs.toArray)方法執行完畢後，進入SparkWordCount的main方法，執行Spark應用程式，如下圖 
至此，正式完成Spark應用程式執行的提交。
轉載： http://blog.csdn.net/lovehuangjiaju/article/details/49123975
相關文章
- Spark修煉之道（進階篇）——Spark入門到精通：第一節 Spark 1.5.0叢集搭建Spark
- Spark修煉之道（進階篇）——Spark入門到精通：第四節 Spark程式設計模型（一)Spark程式設計模型
- Spark修煉之道（進階篇）——Spark入門到精通：第五節 Spark程式設計模型（二)Spark程式設計模型
- Spark修煉之道（進階篇）——Spark入門到精通：第六節 Spark程式設計模型（三)Spark程式設計模型
- Spark修煉之道（進階篇）——Spark入門到精通：第七節 Spark執行原理Spark
- Spark修煉之道（進階篇）——Spark入門到精通：第八節 Spark SQL與DataFrame（一)SparkSQL
- Spark修煉之道（進階篇）——Spark入門到精通：第二節 Hadoop、Spark生成圈簡介SparkHadoop
- Spark修煉之道（進階篇）——Spark入門到精通：第三節 Spark Intellij IDEA開發環境搭建SparkIntelliJIdea開發環境
- Spark on Yarn 任務提交流程原始碼分析SparkYarn原始碼
- Spark文件閱讀之一：Spark OverviewSparkView
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第七節：程式管理SparkLinux大資料
- 【Spark篇】--Spark中Standalone的兩種提交模式Spark模式
- IDEA開發Spark應用並提交本地Spark 2.1.0 standIdeaSpark
- Spark效能優化指南：高階篇Spark優化
- Spark job分配流程原始碼分析Spark原始碼
- 【Spark篇】---Spark初始Spark
- 【Spark篇】---Spark中yarn模式兩種提交任務方式SparkYarn模式
- Spark3.0YarnCluster模式任務提交流程原始碼分析SparkYarn模式原始碼
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第八節：網路管理SparkLinux大資料
- spark 原始碼分析之十九 -- Stage的提交Spark原始碼
- spark原始碼之任務提交過程Spark原始碼
- Spark1.0.0 應用程式部署工具spark-submitSparkMIT
- 【Spark篇】---SparkStream初始與應用Spark
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第一節、Linux介紹、安裝及使用初步SparkLinux大資料
- 【Spark篇】---Spark中資源和任務排程原始碼分析與資源配置引數應用Spark原始碼
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第九節：Shell程式設計入門(一)SparkLinux大資料程式設計
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第十節：Shell程式設計入門（二)SparkLinux大資料程式設計
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第三節：使用者和組SparkLinux大資料
- 【spark筆記】在idea用maven匯入spark原始碼Spark筆記IdeaMaven原始碼
- spark核心(下)——job任務提交原始碼解析Spark原始碼
- 【Spark篇】---Spark中Master-HA和historyServer的搭建和應用SparkASTServer
- Spark core篇 Rpc原始碼1SparkRPC原始碼
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第十二節：Shell程式設計入門（四)SparkLinux大資料程式設計
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第十三節：Shell程式設計入門（五)SparkLinux大資料程式設計
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第十四節：Shell程式設計入門（六)SparkLinux大資料程式設計
- Spark原始碼解析-Yarn部署流程（ApplicationMaster）Spark原始碼YarnAPPAST
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第五節：vi、vim編輯器（一）SparkLinux大資料
- Spark修煉之道（基礎篇）——Linux大資料開發基礎：第六節：vi、vim編輯器（二）SparkLinux大資料