Spark修煉之道（高階篇）——Spark原始碼閱讀：第一節 Spark應用程式提交流程

五柳-先生發表於2015-11-15
Spark原始碼

spark-submit 指令碼應用程式提交流程

在執行Spar應用程式時，會將spark應用程式打包後使用spark-submit指令碼提交到Spark中執行，執行提交命令如下：

<code class="hljs d has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">root<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">@sparkmaster</span>:/hadoopLearning/spark-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span>.0-bin-hadoop2.4/bin# 
./spark-submit --master spark:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//sparkmaster:7077 </span>
--<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> SparkWordCount --executor-memory <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>g
 /root/IdeaProjects/SparkWordCount/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">out</span>/artifacts/SparkWord
 Count_jar/SparkWordCount.jar  hdfs:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//ns1/README.md</span>
 hdfs:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//ns1/SparkWordCountResult</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

為弄清楚整個流程，我們先來分析一下spark-submit指令碼，spark-submit指令碼內容如下：

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-shebang" style="color: rgb(0, 102, 102); box-sizing: border-box;">#!/usr/bin/env bash
</span>
SPARK_HOME=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(cd "`dirname "$0"`"/..; pwd)</span>"</span>

<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># disable randomized hash for string in Python 3.3+</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PYTHONHASHSEED=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>

<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#spark-submit最終呼叫的是spark-class指令碼</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#傳入的類是org.apache.spark.deploy.SparkSubmit</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#及其它傳入的引數，如deploy mode、executor-memory等</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exec</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>"</span>/bin/spark-class org.apache.spark.deploy.SparkSubmit <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$@</span>"</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li></ul>

spark-class指令碼會載入spark配置的環境變數資訊、定位依賴包spark-assembly-1.5.0-hadoop2.4.0.jar檔案（以spark1.5.0為例）等，然後再呼叫org.apache.spark.launcher.Main正式啟動Spark應用程式的執行，具體如下：

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Figure out where Spark is installed</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#定位SAPRK_HOME目錄</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> SPARK_HOME=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(cd "`dirname "$0"`"/..; pwd)</span>"</span>

<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#載入load-spark-env.sh，執行環境相關資訊</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#例如配置檔案conf下的spark-env.sh等</span>
. <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>"</span>/bin/load-spark-env.sh

<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Find the java binary</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># 定位JAVA_HOME目錄</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ -n <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${JAVA_HOME}</span>"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
  RUNNER=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${JAVA_HOME}</span>/bin/java"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ `command -v java` ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
    RUNNER=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"java"</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span>
    <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"JAVA_HOME is not set"</span> >&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exit</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>

<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Find assembly jar</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#定位spark-assembly-1.5.0-hadoop2.4.0.jar檔案（以spark1.5.0為例）</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#這意味著任務提交時無需將該JAR檔案打包</span>
SPARK_ASSEMBLY_JAR=
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ <span class="hljs-operator" style="box-sizing: border-box;">-f</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/RELEASE"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
  ASSEMBLY_DIR=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/lib"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span>
  ASSEMBLY_DIR=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/assembly/target/scala-<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_SCALA_VERSION</span>"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>

num_jars=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(ls -1 "$ASSEMBLY_DIR" | grep "^spark-assembly.*hadoop.*\.jar$" | wc -l)</span>"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$num_jars</span>"</span> <span class="hljs-operator" style="box-sizing: border-box;">-eq</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"0"</span> <span class="hljs-operator" style="box-sizing: border-box;">-a</span> -z <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_ASSEMBLY_JAR</span>"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
  <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Failed to find Spark assembly in <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ASSEMBLY_DIR</span>."</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
  <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"You need to build Spark before running this program."</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exit</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>
ASSEMBLY_JARS=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(ls -1 "$ASSEMBLY_DIR" | grep "^spark-assembly.*hadoop.*\.jar$" || true)</span>"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$num_jars</span>"</span> <span class="hljs-operator" style="box-sizing: border-box;">-gt</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"1"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
  <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Found multiple Spark assembly jars in <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ASSEMBLY_DIR</span>:"</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
  <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ASSEMBLY_JARS</span>"</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
  <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">echo</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Please remove all but one jar."</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>>&<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exit</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>

SPARK_ASSEMBLY_JAR=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${ASSEMBLY_DIR}</span>/<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${ASSEMBLY_JARS}</span>"</span>

LAUNCH_CLASSPATH=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_ASSEMBLY_JAR</span>"</span>

<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># Add the launcher build dir to the classpath if requested.</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> [ -n <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_PREPEND_CLASSES</span>"</span> ]; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span>
  LAUNCH_CLASSPATH=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/launcher/target/scala-<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_SCALA_VERSION</span>/classes:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$LAUNCH_CLASSPATH</span>"</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">fi</span>

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> _SPARK_ASSEMBLY=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_ASSEMBLY_JAR</span>"</span>

<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># The launcher library will print arguments separated by a NULL character, to allow arguments with</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># characters that would be otherwise interpreted by the shell. Read that in a while loop, populating</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># an array that will be used to exec the final command.</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#執行org.apache.spark.launcher.Main作為Spark應用程式的主入口</span>
CMD=()
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">while</span> IFS= <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">read</span> <span class="hljs-operator" style="box-sizing: border-box;">-d</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">''</span> -r ARG; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">do</span>
  CMD+=(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ARG</span>"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">done</span> < <(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$RUNNER</span>"</span> -cp <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$LAUNCH_CLASSPATH</span>"</span> org.apache.spark.launcher.Main <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$@</span>"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exec</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">${CMD[@]}</span>"</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li></ul>

從上述程式碼中，可以看到，通過org.apache.spark.launcher.Main類啟動org.apache.spark.deploy.SparkSubmit的執行，SparkSubmit部分原始碼如下：

<code class="hljs php has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//SparkSubmit Main方法</span>
def main(args: <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">Array</span>[String]): Unit = {
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//任務提交時設定的引數，見圖2</span>
    val appArgs = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkSubmitAarguments(args)
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (appArgs.verbose) {
      <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
      printStream.println(appArgs)
      <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>
    }
    appArgs.action match {
      <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//任務提交時，執行submit(appArgs)</span>
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> SparkSubmitAction.SUBMIT => submit(appArgs)
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> SparkSubmitAction.KILL => kill(appArgs)
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)
    }
  }
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li></ul>

這裡寫圖片描述 
圖1 appArgs = new SparkSubmitAarguments(args)引數

進入submit方法：

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">  <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
   * Submit the application using the provided parameters.
   *
   * This runs in two steps. First, we prepare the launch environment by setting up
   * the appropriate classpath, system properties, and application arguments for
   * running the child main class based on the cluster manager and the deploy mode.
   * Second, we use this launch environment to invoke the main method of the child
   * main class.
   */</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> submit(args: SparkSubmitArguments): Unit = {
   <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//執行引數等資訊，見圖2</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> (childArgs, childClasspath, sysProps, childMainClass) = prepareSubmitEnvironment(args)
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//定義在submit方法中的方法doRunMain()</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> doRunMain(): Unit = {
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (args.proxyUser != <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> proxyUser = UserGroupInformation.createProxyUser(args.proxyUser,
          UserGroupInformation.getCurrentUser())
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
          proxyUser.doAs(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> PrivilegedExceptionAction[Unit]() {
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">override</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> run(): Unit = {
              <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//執行runMain方法</span>
              runMain(childArgs, childClasspath, sysProps, childMainClass, args.verbose)
            }
          })
        } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> {
          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: Exception =>
            <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Hadoop's AuthorizationException suppresses the exception's stack trace, which</span>
            <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// makes the message printed to the output by the JVM not very helpful. Instead,</span>
            <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// detect exceptions with empty stack traces here, and treat them differently.</span>
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (e.getStackTrace().length == <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) {
              <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
              printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"ERROR: ${e.getClass().getName()}: ${e.getMessage()}"</span>)
              <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>
              exitFn(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)
            } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {
              <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> e
            }
        }
      } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//執行runMain方法</span>
        runMain(childArgs, childClasspath, sysProps, childMainClass, args.verbose)
      }
    }

     <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// In standalone cluster mode, there are two submission gateways:</span>
     <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//   (1) The traditional Akka gateway using o.a.s.deploy.Client as a wrapper</span>
     <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//   (2) The new REST-based gateway introduced in Spark 1.3</span>
     <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// The latter is the default behavior as of Spark 1.3, but Spark submit will fail over</span>
     <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// to use the legacy gateway if the master endpoint turns out to be not a REST server.</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (args.isStandaloneCluster && args.useRest) {
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
        printStream.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Running Spark using the REST application submission protocol."</span>)
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//呼叫submit方法中的doRunMain方法</span>
        doRunMain()
      } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> {
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Fail over to use the legacy submission gateway</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: SubmitRestConnectionException =>
          printWarning(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Master endpoint ${args.master} was not a REST server. "</span> +
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Falling back to legacy submission gateway instead."</span>)
          args.useRest = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>
          submit(args)
      }
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// In all other modes, just run the main class as prepared</span>
    } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {
       <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//呼叫submit方法中的doRunMain方法</span>
      doRunMain()
    }
  }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li></ul>

這裡寫圖片描述 
圖2 任務提交時設定的引數，

從上面的程式碼可以看到，最終呼叫的是runMain方法，其原始碼如下：

<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
   * Run the main method of the child class using the provided launch environment.
   *
   * Note that this main class will not be the one provided by the user if we're
   * running cluster deploy mode or python applications.
   */</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> runMain(
      childArgs: Seq[String],
      childClasspath: Seq[String],
      sysProps: Map[String, String],
      childMainClass: String,
      verbose: Boolean): Unit = {
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (verbose) {
      printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Main class:\n$childMainClass"</span>)
      printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Arguments:\n${childArgs.mkString("</span>\n<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")}"</span>)
      printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"System properties:\n${sysProps.mkString("</span>\n<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")}"</span>)
      printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Classpath elements:\n${childClasspath.mkString("</span>\n<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")}"</span>)
      printStream.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\n"</span>)
    }
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> loader =
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (sysProps.getOrElse(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark.driver.userClassPathFirst"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"false"</span>).toBoolean) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> ChildFirstURLClassLoader(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Array[URL](<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),
          Thread.currentThread.getContextClassLoader)
      } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> MutableURLClassLoader(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Array[URL](<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),
          Thread.currentThread.getContextClassLoader)
      }
    Thread.currentThread.setContextClassLoader(loader)

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (jar <- childClasspath) {
      addJarToClasspath(jar, loader)
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> ((key, value) <- sysProps) {
      System.setProperty(key, value)
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">var</span> mainClass: Class[_] = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
      <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//複用反射載入childMainClass，這裡為SparkWordCount</span>
      mainClass = Utils.classForName(childMainClass)
    } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> {
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: ClassNotFoundException =>
        e.printStackTrace(printStream)
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (childMainClass.contains(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"thriftserver"</span>)) {
          <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:off println</span>
          printStream.println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Failed to load main class $childMainClass."</span>)
          printStream.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"You need to build Spark with -Phive and -Phive-thriftserver."</span>)
          <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// scalastyle:on println</span>
        }
        System.exit(CLASS_NOT_FOUND_EXIT_STATUS)
    }

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// SPARK-4170</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (classOf[scala.App].isAssignableFrom(mainClass)) {
      printWarning(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Subclasses of scala.App may not work correctly. Use a main() method instead."</span>)
    }

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//呼叫反射機制載入main方法,即SparkWordCount中的main方法</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> mainMethod = mainClass.getMethod(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"main"</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Array[String](<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>).getClass)
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (!Modifier.isStatic(mainMethod.getModifiers)) {
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> IllegalStateException(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"The main method in the given main class must be static"</span>)
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> findCause(t: Throwable): Throwable = t <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">match</span> {
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: UndeclaredThrowableException =>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (e.getCause() != <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>) findCause(e.getCause()) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> e
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: InvocationTargetException =>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (e.getCause() != <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>) findCause(e.getCause()) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> e
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> e: Throwable =>
        e
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
      <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//執行main方法，即執行SparkWordCount</span>
      mainMethod.invoke(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>, childArgs.toArray)
    } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> {
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> t: Throwable =>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> findCause(t)
    }
  }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li></ul>

mainMethod.invoke(null, childArgs.toArray)方法執行完畢後，進入SparkWordCount的main方法，執行Spark應用程式，如下圖 
這裡寫圖片描述 

至此，正式完成Spark應用程式執行的提交。

轉載： http://blog.csdn.net/lovehuangjiaju/article/details/49123975

相關文章