Spark job分配流程原始碼分析

破棉襖發表於2015-10-13
Spark原始碼


        job提交大體執行流程圖：
Spark job分配流程原始碼分析


SparkContext : scheduler建立 

  1. // Create and start the scheduler
  2.     val (sched, ts) = SparkContext.createTaskScheduler(this, master)
  3.     _schedulerBackend = sched
  4.     _taskScheduler = ts
  5.     _dagScheduler = new DAGScheduler(this)
  6.     _heartbeatReceiver.ask[Boolean](TaskSchedulerIsSet)
SparkContext : createTaskScheduler()方法，主要透過傳入的Master URL來生成Scheduler 和 SchedulerBackend

  1. private def createTaskScheduler(
  2.                                    sc: SparkContext,
  3.                                    master: String): (SchedulerBackend, TaskScheduler) = {
  4.     // Regular expression used for local[N] and local[*] master formats
  5.     val LOCAL_N_REGEX = """local\[([0-9]+|\*)\]""".r
  6.     // Regular expression for local[N, maxRetries], used in tests with failing tasks
  7.     val LOCAL_N_FAILURES_REGEX = """local\[([0-9]+|\*)\s*,\s*([0-9]+)\]""".r
  8.     // Regular expression for simulating a Spark cluster of [N, cores, memory] locally
  9.     val LOCAL_CLUSTER_REGEX = """local-cluster\[\s*([0-9]+)\s*,\s*([0-9]+)\s*,\s*([0-9]+)\s*]""".r
  10.     // Regular expression for connecting to Spark deploy clusters
  11.     val SPARK_REGEX = """spark://(.*)""".r
  12.     // Regular expression for connection to Mesos cluster by mesos:// or zk:// url
  13.     val MESOS_REGEX = """(mesos|zk)://.*""".r
  14.     // Regular expression for connection to Simr cluster
  15.     val SIMR_REGEX = """simr://(.*)""".r

  17.     // When running locally, don't try to re-execute tasks on failure.
  18.     val MAX_LOCAL_TASK_FAILURES = 1

  20.     master match {
  21.       case "local" =>
  22.         val scheduler = new TaskSchedulerImpl(sc, MAX_LOCAL_TASK_FAILURES, isLocal = true)
  23.         val backend = new LocalBackend(sc.getConf, scheduler, 1)
  24.         scheduler.initialize(backend)
  25.         (backend, scheduler)

  27.       case LOCAL_N_REGEX(threads) =>
  28.         def localCpuCount: Int = Runtime.getRuntime.availableProcessors()
  29.         // local[*] estimates the number of cores on the machine; local[N] uses exactly N threads.
  30.         val threadCount = if (threads == "*") localCpuCount else threads.toInt
  31.         if (threadCount <= 0) {
  32.           throw new SparkException(s"Asked to run locally with $threadCount threads")
  33.         }
  34.         val scheduler = new TaskSchedulerImpl(sc, MAX_LOCAL_TASK_FAILURES, isLocal = true)
  35.         val backend = new LocalBackend(sc.getConf, scheduler, threadCount)
  36.         scheduler.initialize(backend)
  37.         (backend, scheduler)

  39.       case LOCAL_N_FAILURES_REGEX(threads, maxFailures) =>
  40.         def localCpuCount: Int = Runtime.getRuntime.availableProcessors()
  41.         // local[*, M] means the number of cores on the computer with M failures
  42.         // local[N, M] means exactly N threads with M failures
  43.         val threadCount = if (threads == "*") localCpuCount else threads.toInt
  44.         val scheduler = new TaskSchedulerImpl(sc, maxFailures.toInt, isLocal = true)
  45.         val backend = new LocalBackend(sc.getConf, scheduler, threadCount)
  46.         scheduler.initialize(backend)
  47.         (backend, scheduler)

  49.       case SPARK_REGEX(sparkUrl) =>
  50.         val scheduler = new TaskSchedulerImpl(sc)
  51.         val masterUrls = sparkUrl.split(",").map("spark://" + _)
  52.         val backend = new SparkDeploySchedulerBackend(scheduler, sc, masterUrls)
  53.         scheduler.initialize(backend)
  54.         (backend, scheduler)

  56.       case LOCAL_CLUSTER_REGEX(numSlaves, coresPerSlave, memoryPerSlave) =>
  57.         // Check to make sure memory requested <= memoryPerSlave. Otherwise Spark will just hang.
  58.         val memoryPerSlaveInt = memoryPerSlave.toInt
  59.         if (sc.executorMemory > memoryPerSlaveInt) {
  60.           throw new SparkException(
  61.             "Asked to launch cluster with %d MB RAM / worker but requested %d MB/worker".format(
  62.               memoryPerSlaveInt, sc.executorMemory))
  63.         }

  65.         val scheduler = new TaskSchedulerImpl(sc)
  66.         val localCluster = new LocalSparkCluster(
  67.           numSlaves.toInt, coresPerSlave.toInt, memoryPerSlaveInt, sc.conf)
  68.         val masterUrls = localCluster.start()
  69.         val backend = new SparkDeploySchedulerBackend(scheduler, sc, masterUrls)
  70.         scheduler.initialize(backend)
  71.         backend.shutdownCallback = (backend: SparkDeploySchedulerBackend) => {
  72.           localCluster.stop()
  73.         }
  74.         (backend, scheduler)

  76.       case "yarn-standalone" | "yarn-cluster" =>
  77.         if (master == "yarn-standalone") {
  78.           logWarning(
  79.             "\"yarn-standalone\" is deprecated as of Spark 1.0. Use \"yarn-cluster\" instead.")
  80.         }
  81.         val scheduler = try {
  82.           val clazz = Utils.classForName("org.apache.spark.scheduler.cluster.YarnClusterScheduler")
  83.           val cons = clazz.getConstructor(classOf[SparkContext])
  84.           cons.newInstance(sc).asInstanceOf[TaskSchedulerImpl]
  85.         } catch {
  86.           // TODO: Enumerate the exact reasons why it can fail
  87.           // But irrespective of it, it means we cannot proceed !
  88.           case e: Exception => {
  89.             throw new SparkException("YARN mode not available ?", e)
  90.           }
  91.         }
  92.         val backend = try {
  93.           val clazz =
  94.             Utils.classForName("org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend")
  95.           val cons = clazz.getConstructor(classOf[TaskSchedulerImpl], classOf[SparkContext])
  96.           cons.newInstance(scheduler, sc).asInstanceOf[CoarseGrainedSchedulerBackend]
  97.         } catch {
  98.           case e: Exception => {
  99.             throw new SparkException("YARN mode not available ?", e)
  100.           }
  101.         }
  102.         scheduler.initialize(backend)
  103.         (backend, scheduler)

  105.       case "yarn-client" =>
  106.         val scheduler = try {
  107.           val clazz = Utils.classForName("org.apache.spark.scheduler.cluster.YarnScheduler")
  108.           val cons = clazz.getConstructor(classOf[SparkContext])
  109.           cons.newInstance(sc).asInstanceOf[TaskSchedulerImpl]

  111.         } catch {
  112.           case e: Exception => {
  113.             throw new SparkException("YARN mode not available ?", e)
  114.           }
  115.         }

  117.         val backend = try {
  118.           val clazz =
  119.             Utils.classForName("org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend")
  120.           val cons = clazz.getConstructor(classOf[TaskSchedulerImpl], classOf[SparkContext])
  121.           cons.newInstance(scheduler, sc).asInstanceOf[CoarseGrainedSchedulerBackend]
  122.         } catch {
  123.           case e: Exception => {
  124.             throw new SparkException("YARN mode not available ?", e)
  125.           }
  126.         }

  128.         scheduler.initialize(backend)
  129.         (backend, scheduler)

  131.       case mesosUrl @ MESOS_REGEX(_) =>
  132.         MesosNativeLibrary.load()
  133.         val scheduler = new TaskSchedulerImpl(sc)
  134.         val coarseGrained = sc.conf.getBoolean("spark.mesos.coarse", false)
  135.         val url = mesosUrl.stripPrefix("mesos://") // strip scheme from raw Mesos URLs
  136.       val backend = if (coarseGrained) {
  137.           new CoarseMesosSchedulerBackend(scheduler, sc, url, sc.env.securityManager)
  138.         } else {
  139.           new MesosSchedulerBackend(scheduler, sc, url)
  140.         }
  141.         scheduler.initialize(backend)
  142.         (backend, scheduler)

  144.       case SIMR_REGEX(simrUrl) =>
  145.         val scheduler = new TaskSchedulerImpl(sc)
  146.         val backend = new SimrSchedulerBackend(scheduler, sc, simrUrl)
  147.         scheduler.initialize(backend)
  148.         (backend, scheduler)

  150.       case _ =>
  151.         throw new SparkException("Could not parse Master URL: '" + master + "'")
  152.     }
  153.   }
  154. }


RDD的Action操作會呼叫sc的runJob方法
1.SparkContext的runJob方法： 


  1. def runJob[T, U: ClassTag](
  2.       rdd: RDD[T], //此處是具體的 RDD 例項值

  4.       func: (TaskContext, Iterator[T]) => U, //具體的執行的 action 的邏輯 , 如 reduceByKey

  6.       partitions: Seq[Int], //分割槽陣列 , 一個數值從 0 到 partitions.size-1

  8.       //result 的處理邏輯 , 每一個 Task 的處理

  10.       resultHandler: (Int, U) => Unit): Unit = {
  11.     if (stopped.get()) {
  12.       throw new IllegalStateException("SparkContext has been shutdown")
  13.     }
  14.     val callSite = getCallSite
  15.     val cleanedFunc = clean(func)
  16.     logInfo("Starting job: " + callSite.shortForm)
  17.     if (conf.getBoolean("spark.logLineage", false)) {
  18.       logInfo("RDD's recursive dependencies:\n" + rdd.toDebugString)
  19.     }
  20.     dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, resultHandler, localProperties.get)
  21.     progressBar.foreach(_.finishAll())
  22.     rdd.doCheckpoint()
  23.   }
跟進：DAGScheduler

2.DAGScheduler的runjob方法:
  1. def runJob[T, U](
  2.       rdd: RDD[T],
  3.       func: (TaskContext, Iterator[T]) => U,
  4.       partitions: Seq[Int],
  5.       callSite: CallSite,
  6.       resultHandler: (Int, U) => Unit,
  7.       properties: Properties): Unit = {
  8.     val start = System.nanoTime
  9.     val waiter = submitJob(rdd, func, partitions, callSite, resultHandler, properties)
  10.       //等待 job 執行完成。

  12.     waiter.awaitResult() match {
  13.       case JobSucceeded =>
  14.         logInfo("Job %d finished: %s, took %f s".format
  15.           (waiter.jobId, callSite.shortForm, (System.nanoTime - start) / 1e9))
  16.       case JobFailed(exception: Exception) =>
  17.         logInfo("Job %d failed: %s, took %f s".format
  18.           (waiter.jobId, callSite.shortForm, (System.nanoTime - start) / 1e9))
  19.         // SPARK-8644: Include user stack trace in exceptions coming from DAGScheduler.

  21.         val callerStackTrace = Thread.currentThread().getStackTrace.tail
  22.         exception.setStackTrace(exception.getStackTrace ++ callerStackTrace)
  23.         throw exception
  24.     }
  25.   }
繼續進入submitJob：
  1. def submitJob[T, U](
  2.       rdd: RDD[T],
  3.       func: (TaskContext, Iterator[T]) => U,
  4.       partitions: Seq[Int],
  5.       callSite: CallSite,
  6.       resultHandler: (Int, U) => Unit,
  7.       properties: Properties): JobWaiter[U] = {
  8.     // Check to make sure we are not launching a task on a partition that does not exist.

  10.     val maxPartitions = rdd.partitions.length
  11.     partitions.find(p => p >= maxPartitions || p < 0).foreach { p =>
  12.       throw new IllegalArgumentException(
  13.         "Attempting to access a non-existent partition: " + p + ". " +
  14.           "Total number of partitions: " + maxPartitions)
  15.     }

  17.     val jobId = nextJobId.getAndIncrement()
  18.     if (partitions.size == 0) {
  19.       // Return immediately if the job is running 0 tasks

  21.       return new JobWaiter[U](this, jobId, 0, resultHandler)
  22.     }

  24.     assert(partitions.size > 0)
  25.     val func2 = func.asInstanceOf[(TaskContext, Iterator[_]) => _]
  26.     val waiter = new JobWaiter(this, jobId, partitions.size, resultHandler)
  27.     eventProcessLoop.post(JobSubmitted( // 向佇列新增一個 event, 此 event 為 JobSubmitted

  29.       jobId, rdd, func2, partitions.toArray, callSite, waiter,
  30.       SerializationUtils.clone(properties)))
  31.     waiter
  32.   }

submitJob方法將JobSubmitted傳送到佇列（並沒有採用Actor模型  而是1.4版本後重新採用了1.0版本之前的佇列方式），那佇列是在哪消費的呢？ 看下邊;
在DAGScheduler類的最後有這麼一段：

  1. // Start the event thread and register the metrics source at the end of the constructor

  3.   env.metricsSystem.registerSource(metricsSource)
  4.   eventProcessLoop.start()
這段程式碼是寫在方法體外的，如果這是一個java類，則這顯然是不合乎語法的。這裡涉及一個scala的特性：在一個類裡，你可以隨意的寫一些指令碼，或者說是程式碼片段更加形象，這些指令碼會在物件例項化時被執行。

DAGSchedulerEventProcessLoop 繼承 EventLoop，看一下EventLoop的start方法：
      
  1.   //開一個執行緒不斷消費佇列 呼叫onReceive()方法

  3.   private val eventThread = new Thread(name) {
  4.     setDaemon(true)

  6.     override def run(): Unit = {
  7.       try {
  8.         while (!stopped.get) {
  9.           val event = eventQueue.take()
  10.           try {
  11.             onReceive(event)
  12.           } catch {
  13.             case NonFatal(e) => {
  14.               try {
  15.                 onError(e)
  16.               } catch {
  17.                 case NonFatal(e) => logError("Unexpected error in " + name, e)
  18.               }
  19.             }
  20.           }
  21.         }
  22.       } catch {
  23.         case ie: InterruptedException => // exit even if eventQueue is not empty

  25.         case NonFatal(e) => logError("Unexpected error in " + name, e)
  26.       }
  27.     }

  29.   }

  31.   def start(): Unit = {
  32.     if (stopped.get) {
  33.       throw new IllegalStateException(name + " has already been stopped")
  34.     }
  35.     // Call onStart before starting the event thread to make sure it happens before onReceive

  37.     onStart()
  38.     eventThread.start()
  39.   }
由此可知在DAGScheduler初始化時會起一個執行緒去進行佇列消費。
接下來看onReceive()方法：
  1. /**
  2.    * The main event loop of the DAG scheduler.
  3.    */
  4.   override def onReceive(event: DAGSchedulerEvent): Unit = {
  5.     val timerContext = timer.time()
  6.     try {
  7.       doOnReceive(event)
  8.     } finally {
  9.       timerContext.stop()
  10.     }
  11.   }

  1. private def doOnReceive(event: DAGSchedulerEvent): Unit = event match {
  2.     case JobSubmitted(jobId, rdd, func, partitions, callSite, listener, properties) =>
  3.       dagScheduler.handleJobSubmitted(jobId, rdd, func, partitions, callSite, listener, properties)

  5.     case MapStageSubmitted(jobId, dependency, callSite, listener, properties) =>
  6.       dagScheduler.handleMapStageSubmitted(jobId, dependency, callSite, listener, properties)

  8.     case StageCancelled(stageId) =>
  9.       dagScheduler.handleStageCancellation(stageId)

  11.     case JobCancelled(jobId) =>
  12.       dagScheduler.handleJobCancellation(jobId)

  14.     case JobGroupCancelled(groupId) =>
  15.       dagScheduler.handleJobGroupCancelled(groupId)

  17.     case AllJobsCancelled =>
  18.       dagScheduler.doCancelAllJobs()

  20.     case ExecutorAdded(execId, host) =>
  21.       dagScheduler.handleExecutorAdded(execId, host)

  23.     case ExecutorLost(execId) =>
  24.       dagScheduler.handleExecutorLost(execId, fetchFailed = false)

  26.     case BeginEvent(task, taskInfo) =>
  27.       dagScheduler.handleBeginEvent(task, taskInfo)

  29.     case GettingResultEvent(taskInfo) =>
  30.       dagScheduler.handleGetTaskResult(taskInfo)

  32.     case completion @ CompletionEvent(task, reason, _, _, taskInfo, taskMetrics) =>
  33.       dagScheduler.handleTaskCompletion(completion)

  35.     case TaskSetFailed(taskSet, reason, exception) =>
  36.       dagScheduler.handleTaskSetFailed(taskSet, reason, exception)

  38.     case ResubmitFailedStages =>
  39.       dagScheduler.resubmitFailedStages()
  40.   }

進入handleJobSubmitted方法：
  1. private[scheduler] def handleJobSubmitted(jobId: Int,
  2.       finalRDD: RDD[_],
  3.       func: (TaskContext, Iterator[_]) => _,
  4.       partitions: Array[Int],
  5.       callSite: CallSite,
  6.       listener: JobListener,
  7.       properties: Properties) {
  8.     var finalStage: ResultStage = null
  9.     try {
  10.       // New stage creation may throw an exception if, for example, jobs are run on a

  12.       // HadoopRDD whose underlying HDFS files have been deleted.

  14.       //生成一個 finalStage, 每一個 JOB 都有一個 finalStage, 根據 job 劃分出不同的 stage , RDD提交過來的肯定為最後一個RDD 所以此處封裝為ResultStage 
  15.       //RDD中：  def count(): Long = sc.runJob(this, Utils.getIteratorSize _).sum

  17.       finalStage = newResultStage(finalRDD, func, partitions, jobId, callSite)
  18.     } catch {
  19.       case e: Exception =>
  20.         logWarning("Creating new stage failed due to exception - job: " + jobId, e)
  21.         listener.jobFailed(e)
  22.         return
  23.     }

  25.     val job = new ActiveJob(jobId, finalStage, callSite, listener, properties)
  26.     clearCacheLocs()
  27.     logInfo("Got job %s (%s) with %d output partitions".format(
  28.       job.jobId, callSite.shortForm, partitions.length))
  29.     logInfo("Final stage: " + finalStage + " (" + finalStage.name + ")")
  30.     logInfo("Parents of final stage: " + finalStage.parents)
  31.     logInfo("Missing parents: " + getMissingParentStages(finalStage))

  33.     val jobSubmissionTime = clock.getTimeMillis()
  34.     jobIdToActiveJob(jobId) = job
  35.     activeJobs += job
  36.     finalStage.resultOfJob = Some(job)
  37.     val stageIds = jobIdToStageIds(jobId).toArray
  38.     val stageInfos = stageIds.flatMap(id => stageIdToStage.get(id).map(_.latestInfo))
  39.     listenerBus.post(
  40.       SparkListenerJobStart(job.jobId, jobSubmissionTime, stageInfos, properties))
  41.     submitStage(finalStage)

  43.     submitWaitingStages()
  44.   }
進入submitStage方法：
  1. /** Submits stage, but first recursively submits any missing parents. */
  2.   //提交階段，但首先遞迴提交的父母                        首次進來為ResultStage,在此方法中遞迴提交父RDD組成的ShuffleMapStage
  3.   private def submitStage(stage: Stage) {
  4.     val jobId = activeJobForStage(stage)
  5.     if (jobId.isDefined) {
  6.       logDebug("submitStage(" + stage + ")")
  7.       if (!waitingStages(stage) && !runningStages(stage) && !failedStages(stage)) {
  8.         val missing = getMissingParentStages(stage).sortBy(_.id)                                         //獲取父Stages
  9.         logDebug("missing: " + missing)
  10.         if (missing.isEmpty) {
  11.           logInfo("Submitting " + stage + " (" + stage.rdd + "), which has no missing parents")
  12.           submitMissingTasks(stage, jobId.get)
  13.         } else {
  14.           for (parent <- missing) {
  15.             submitStage(parent)
  16.           }
  17.           waitingStages += stage
  18.         }
  19.       }
  20.     } else {
  21.       abortStage(stage, "No active job for stage " + stage.id, None)
  22.     }
  23.   }


繼續進入submitMissingTasks方法：
  1. /** Called when stage's parents are available and we can now do its task. */
  2.   // taskScheduler.submitTasks 提交

  4.   private def submitMissingTasks(stage: Stage, jobId: Int) {
  5.     logDebug("submitMissingTasks(" + stage + ")")
  6.     // Get our pending tasks and remember them in our pendingTasks entry

  8.     stage.pendingTasks.clear()

  10.     // First figure out the indexes of partition ids to compute.

  12.     val (allPartitions: Seq[Int], partitionsToCompute: Seq[Int]) = {
  13.       stage match {                                                            //stage分為兩種型別 shuffleMapStage和ResultStage
  14.         case stage: ShuffleMapStage =>
  15.           val allPartitions = 0 until stage.numPartitions
  16.           val filteredPartitions = allPartitions.filter { id => stage.outputLocs(id).isEmpty }
  17.           (allPartitions, filteredPartitions)
  18.         case stage: ResultStage =>
  19.           val job = stage.resultOfJob.get
  20.           val allPartitions = 0 until job.numPartitions
  21.           val filteredPartitions = allPartitions.filter { id => !job.finished(id) }
  22.           (allPartitions, filteredPartitions)
  23.       }
  24.     }

  26.     // Create internal accumulators if the stage has no accumulators initialized.

  28.     // Reset internal accumulators only if this stage is not partially submitted

  30.     // Otherwise, we may override existing accumulator values from some tasks

  32.     if (stage.internalAccumulators.isEmpty || allPartitions == partitionsToCompute) {
  33.       stage.resetInternalAccumulators()
  34.     }

  36.     val properties = jobIdToActiveJob.get(stage.firstJobId).map(_.properties).orNull

  38.     runningStages += stage
  39.     // SparkListenerStageSubmitted should be posted before testing whether tasks are

  41.     // serializable. If tasks are not serializable, a SparkListenerStageCompleted event

  43.     // will be posted, which should always come after a corresponding SparkListenerStageSubmitted

  45.     // event.

  47.     outputCommitCoordinator.stageStart(stage.id)
  48.     val taskIdToLocations = try {
  49.       stage match {
  50.         case s: ShuffleMapStage =>
  51.           partitionsToCompute.map { id => (id, getPreferredLocs(stage.rdd, id))}.toMap
  52.         case s: ResultStage =>
  53.           val job = s.resultOfJob.get
  54.           partitionsToCompute.map { id =>
  55.             val p = s.partitions(id)
  56.             (id, getPreferredLocs(stage.rdd, p))
  57.           }.toMap
  58.       }
  59.     } catch {
  60.       case NonFatal(e) =>
  61.         stage.makeNewStageAttempt(partitionsToCompute.size)
  62.         listenerBus.post(SparkListenerStageSubmitted(stage.latestInfo, properties))
  63.         abortStage(stage, s"Task creation failed: $e\n${e.getStackTraceString}", Some(e))
  64.         runningStages -= stage
  65.         return
  66.     }

  68.     stage.makeNewStageAttempt(partitionsToCompute.size, taskIdToLocations.values.toSeq)
  69.     listenerBus.post(SparkListenerStageSubmitted(stage.latestInfo, properties))

  71.     // TODO: Maybe we can keep the taskBinary in Stage to avoid serializing it multiple times.

  73.     // Broadcasted binary for the task, used to dispatch tasks to executors. Note that we broadcast

  75.     // the serialized copy of the RDD and for each task we will deserialize it, which means each

  77.     // task gets a different copy of the RDD. This provides stronger isolation between tasks that

  79.     // might modify state of objects referenced in their closures. This is necessary in Hadoop

  81.     // where the JobConf/Configuration object is not thread-safe.

  83.     var taskBinary: Broadcast[Array[Byte]] = null
  84.     try {
  85.       // For ShuffleMapTask, serialize and broadcast (rdd, shuffleDep).

  87.       // For ResultTask, serialize and broadcast (rdd, func).

  89.       val taskBinaryBytes: Array[Byte] = stage match {
  90.         case stage: ShuffleMapStage =>
  91.           closureSerializer.serialize((stage.rdd, stage.shuffleDep): AnyRef).array()
  92.         case stage: ResultStage =>
  93.           closureSerializer.serialize((stage.rdd, stage.func): AnyRef).array()
  94.       }

  96.       taskBinary = sc.broadcast(taskBinaryBytes)
  97.     } catch {
  98.       // In the case of a failure during serialization, abort the stage.

  100.       case e: NotSerializableException =>
  101.         abortStage(stage, "Task not serializable: " + e.toString, Some(e))
  102.         runningStages -= stage

  104.         // Abort execution

  106.         return
  107.       case NonFatal(e) =>
  108.         abortStage(stage, s"Task serialization failed: $e\n${e.getStackTraceString}", Some(e))
  109.         runningStages -= stage
  110.         return
  111.     }

  113.     val tasks: Seq[Task[_]] = try {
  114.       stage match {
  115.         case stage: ShuffleMapStage =>
  116.           partitionsToCompute.map { id =>
  117.             val locs = taskIdToLocations(id)
  118.             val part = stage.rdd.partitions(id)
  119.             new ShuffleMapTask(stage.id, stage.latestInfo.attemptId,
  120.               taskBinary, part, locs, stage.internalAccumulators)
  121.           }

  123.         case stage: ResultStage =>
  124.           val job = stage.resultOfJob.get
  125.           partitionsToCompute.map { id =>
  126.             val p: Int = stage.partitions(id)
  127.             val part = stage.rdd.partitions(p)
  128.             val locs = taskIdToLocations(id)
  129.             new ResultTask(stage.id, stage.latestInfo.attemptId,
  130.               taskBinary, part, locs, id, stage.internalAccumulators)
  131.           }
  132.       }
  133.     } catch {
  134.       case NonFatal(e) =>
  135.         abortStage(stage, s"Task creation failed: $e\n${e.getStackTraceString}", Some(e))
  136.         runningStages -= stage
  137.         return
  138.     }

  140.     if (tasks.size > 0) {
  141.       logInfo("Submitting " + tasks.size + " missing tasks from " + stage + " (" + stage.rdd + ")")
  142.       stage.pendingTasks ++= tasks
  143.       logDebug("New pending tasks: " + stage.pendingTasks)
  144.       //把 stage 根據 partition 生成 TaskSet, 透過 TaskScheduler 提交 Task

  146.       taskScheduler.submitTasks(new TaskSet(
  147.         tasks.toArray, stage.id, stage.latestInfo.attemptId, stage.firstJobId, properties))
  148.       stage.latestInfo.submissionTime = Some(clock.getTimeMillis())
  149.     } else {
  150.       // Because we posted SparkListenerStageSubmitted earlier, we should mark

  152.       // the stage as completed here in case there are no tasks to run

  154.       markStageAsFinished(stage, None)

  156.       val debugString = stage match {
  157.         case stage: ShuffleMapStage =>
  158.           s"Stage ${stage} is actually done; " +
  159.             s"(available: ${stage.isAvailable}," +
  160.             s"available outputs: ${stage.numAvailableOutputs}," +
  161.             s"partitions: ${stage.numPartitions})"
  162.         case stage : ResultStage =>
  163.           s"Stage ${stage} is actually done; (partitions: ${stage.numPartitions})"
  164.       }
  165.       logDebug(debugString)
  166.     }
  167.   }

此處不深入研究該方法看一下taskScheduler.submitTasks方法：
  1. override def submitTasks(taskSet: TaskSet) {
  2.     val tasks = taskSet.tasks
  3.     logInfo("Adding task set " + taskSet.id + " with " + tasks.length + " tasks")
  4.     this.synchronized {

  6.       //生成一個 TaskSetManager 例項，TaskSetManager負責監控該TaskSet的所有Task任務，有任務完成後，taskScheduler會從該Manager中刪除task

  8.       val manager = createTaskSetManager(taskSet, maxTaskFailures)
  9.       val stage = taskSet.stageId
  10.       val stageTaskSets =
  11.         taskSetsByStageIdAndAttempt.getOrElseUpdate(stage, new HashMap[Int, TaskSetManager])
  12.       stageTaskSets(taskSet.stageAttemptId) = manager
  13.       val conflictingTaskSet = stageTaskSets.exists { case (_, ts) =>
  14.         ts.taskSet != taskSet && !ts.isZombie
  15.       }
  16.       if (conflictingTaskSet) {
  17.         throw new IllegalStateException(s"more than one active taskSet for stage $stage:" +
  18.           s" ${stageTaskSets.toSeq.map{_._2.taskSet.id}.mkString(",")}")
  19.       }
  20.       schedulableBuilder.addTaskSetManager(manager, manager.taskSet.properties)                //將manager加入到rootPool池中，在初始化schedulableBuilder時賦予rootPool池（FIFO、FAIR）


  22.       if (!isLocal && !hasReceivedTask) {
  23.         starvationTimer.scheduleAtFixedRate(new TimerTask() {
  24.           override def run() {
  25.             if (!hasLaunchedTask) {
  26.               logWarning("Initial job has not accepted any resources; " +
  27.                 "check your cluster UI to ensure that workers are registered " +
  28.                 "and have sufficient resources")
  29.             } else {
  30.               this.cancel()
  31.             }
  32.           }
  33.         }, STARVATION_TIMEOUT_MS, STARVATION_TIMEOUT_MS)
  34.       }
  35.       hasReceivedTask = true
  36.     }
  37.     //透過 backend 發起執行訊息 ,backend 是 SchedulerBackend 的具體實現，在 yarn-cluster 模式為 CoarseGrainedSchedulerBackend


  40.     backend.reviveOffers()
  41.   }


CoarseGrainedSchedulerBackend：

  1. override def reviveOffers() {
  2.     driverEndpoint.send(ReviveOffers)
  3.   }
可見reviveOffers方法是向schedulerBackend傳送了一個ReviveOffers訊息。schedulerBackend為Actor模型看一下start方法，taskScheduler中初始化時會呼叫start()方法：

  1. override def start() {
  2.     val properties = new ArrayBuffer[(String, String)]
  3.     for ((key, value) <- scheduler.sc.conf.getAll) {
  4.       if (key.startsWith("spark.")) {
  5.         properties += ((key, value))
  6.       }
  7.     }

  9.     // TODO (prashant) send conf instead of properties   
  10.     // 初始化話 endpoint 用來通訊
  11.     driverEndpoint = rpcEnv.setupEndpoint(ENDPOINT_NAME, createDriverEndpoint(properties))
  12.   }

    //建立一個Actor
  1.   protected def createDriverEndpoint(properties: Seq[(String, String)]): DriverEndpoint = {
  2.     new DriverEndpoint(rpcEnv, properties)
  3.   }




看一下receive方法：

  1. override def receive: PartialFunction[Any, Unit] = {
  2.       case StatusUpdate(executorId, taskId, state, data) =>
  3.         scheduler.statusUpdate(taskId, state, data.value)
  4.         if (TaskState.isFinished(state)) {
  5.           executorDataMap.get(executorId) match {
  6.             case Some(executorInfo) =>
  7.               executorInfo.freeCores += scheduler.CPUS_PER_TASK
  8.               makeOffers(executorId)
  9.             case None =>
  10.               // Ignoring the update since we don't know about the executor.
  11.               logWarning(s"Ignored task status update ($taskId state $state) " +
  12.                 s"from unknown executor with ID $executorId")
  13.           }
  14.         }

  16.       case ReviveOffers =>                          //在初始化時會每隔一定時間（spark.scheduler.revive.interval, 預設為1s）進行一次排程（給自身傳送ReviveOffers訊息， 進行呼叫makeOffers進行排程）。
  17.         makeOffers()

  19.       case KillTask(taskId, executorId, interruptThread) =>
  20.         executorDataMap.get(executorId) match {
  21.           case Some(executorInfo) =>
  22.             executorInfo.executorEndpoint.send(KillTask(taskId, executorId, interruptThread))
  23.           case None =>
  24.             // Ignoring the task kill since the executor is not registered.
  25.             logWarning(s"Attempted to kill task $taskId for unknown executor $executorId.")
  26.         }

  28.     }
看紅色部分進入makeOffers（）方法：
  1. // Make fake resource offers on all executors
  2.     private def makeOffers() {
  3.       // Filter out executors under killing
  4.       val activeExecutors = executorDataMap.filterKeys(!executorsPendingToRemove.contains(_))         //過濾出executor
  5.       val workOffers = activeExecutors.map { case (id, executorData) =>
  6.         new WorkerOffer(id, executorData.executorHost, executorData.freeCores)                        //轉為seq
  7.       }.toSeq                                                    
  8.       launchTasks(scheduler.resourceOffers(workOffers))
  9.     }
重點是TaskScheduler的resourceOffers()方法：

  1. def resourceOffers(offers: Seq[WorkerOffer]): Seq[Seq[TaskDescription]] = synchronized { //SchedulerBackend中呼叫 封裝TaskDescription 傳送給Executor端執行
  2.     // Mark each slave as alive and remember its hostname
  3.     // Also track if new executor is added
  4.     var newExecAvail = false
  5.     // 遍歷worker提供的資源，更新executor相關的對映
  6.     for (o <- offers) {
  7.       executorIdToHost(o.executorId) = o.host
  8.       activeExecutorIds += o.executorId
  9.       if (!executorsByHost.contains(o.host)) {
  10.         executorsByHost(o.host) = new HashSet[String]()
  11.         executorAdded(o.executorId, o.host)
  12.         newExecAvail = true
  13.       }
  14.       for (rack <- getRackForHost(o.host)) {
  15.         hostsByRack.getOrElseUpdate(rack, new HashSet[String]()) += o.host
  16.       }
  17.     }
  18.     // 從worker當中隨機選出一些來，防止任務都堆在一個機器上
  19.     // Randomly shuffle offers to avoid always placing tasks on the same set of workers.
  20.     val shuffledOffers = Random.shuffle(offers) //將offers集合打散順序
  21.     // Build a list of tasks to assign to each worker.
  22.     val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](o.cores))
  23.     val availableCpus = shuffledOffers.map(o => o.cores).toArray
  24.     // getSortedTask函式對taskset進行排序
  25.     val sortedTaskSets = rootPool.getSortedTaskSetQueue //排序佇列 獲取到一組 TaskSetManager ：ArrayBuffer[TaskSetManager]
  26.     //按就近原則進行Task排程
  27.     for (taskSet <- sortedTaskSets) {
  28.       logDebug("parentName: %s, name: %s, runningTasks: %s".format(
  29.         taskSet.parent.name, taskSet.name, taskSet.runningTasks))
  30.       if (newExecAvail) {
  31.         taskSet.executorAdded()
  32.       }
  33.     }

  35.     // Take each TaskSet in our scheduling order, and then offer it each node in increasing order
  36.     // of locality levels so that it gets a chance to launch local tasks on all of them.
  37.     // NOTE: the preferredLocality order: PROCESS_LOCAL, NODE_LOCAL, NO_PREF, RACK_LOCAL, ANY
  38.     var launchedTask = false
  39.     for (taskSet <- sortedTaskSets; maxLocality <- taskSet.myLocalityLevels) {
  40.       do {
  41.         launchedTask = resourceOfferSingleTaskSet(
  42.             taskSet, maxLocality, shuffledOffers, availableCpus, tasks)
  43.       } while (launchedTask)
  44.     }

  46.     if (tasks.size > 0) {
  47.       hasLaunchedTask = true
  48.     }
  49.     return tasks
  50.   }
看一下resourceOfferSingleTaskSet():

  1. private def resourceOfferSingleTaskSet(
  2.       taskSet: TaskSetManager,
  3.       maxLocality: TaskLocality,
  4.       shuffledOffers: Seq[WorkerOffer],
  5.       availableCpus: Array[Int],
  6.       tasks: Seq[ArrayBuffer[TaskDescription]]) : Boolean = {
  7.     var launchedTask = false
  8.     for (i <- 0 until shuffledOffers.size) {  // 遍歷所有Executor
  9.       val execId = shuffledOffers(i).executorId
  10.       val host = shuffledOffers(i).host
  11.       if (availableCpus(i) >= CPUS_PER_TASK) { // >1
  12.         try {
  13.           for (task <- taskSet.resourceOffer(execId, host, maxLocality)) {    // 根據executor的資訊返回一個最合適的Task
  14.             // resourceOffer ：
  15.             //根據TaskScheduler所提供的單個Resource資源包括host，executor和locality的要求返回一個合適的Task。
  16.             // TaskSetManager內部會根據上一個任務成功提交的時間，自動調整自身的Locality匹配策略，
  17.             // 如果上一次成功提交任務的時間間隔很長，則降低對Locality的要求（例如從最差要求Process Local降低為最差要求Node Local），
  18.             // 反之則提高對Locality的要求。這一動態調整Locality策略基本可以理解為是為了提高任務在最佳Locality的情況下得到執行的機會，
  19.             // 因為Resource資源可能是在短期內分批提供給TaskSetManager的，動態調整Locality門檻有助於改善整體的Locality分佈情況。
  20.             tasks(i) += task
  21.             val tid = task.taskId
  22.             taskIdToTaskSetManager(tid) = taskSet
  23.             taskIdToExecutorId(tid) = execId
  24.             executorsByHost(host) += execId
  25.             availableCpus(i) -= CPUS_PER_TASK
  26.             assert(availableCpus(i) >= 0)
  27.             launchedTask = true
  28.           }
  29.         } catch {
  30.           case e: TaskNotSerializableException =>
  31.             logError(s"Resource offer failed, task set ${taskSet.name} was not serializable")
  32.             // Do not offer resources for this task, but don't throw an error to allow other
  33.             // task sets to be submitted.
  34.             return launchedTask
  35.         }
  36.       }
  37.     }
  38.     return launchedTask
  39.   }


launchTasks()方法：
  1. // Launch tasks returned by a set of resource offers
  2.     private def launchTasks(tasks: Seq[Seq[TaskDescription]]) {
  3.       for (task <- tasks.flatten) {
  4.         val serializedTask = ser.serialize(task)
  5.         if (serializedTask.limit >= akkaFrameSize - AkkaUtils.reservedSizeBytes) {
  6.           scheduler.taskIdToTaskSetManager.get(task.taskId).foreach { taskSetMgr =>
  7.             try {
  8.               var msg = "Serialized task %s:%d was %d bytes, which exceeds max allowed: " +
  9.                 "spark.akka.frameSize (%d bytes) - reserved (%d bytes). Consider increasing " +
  10.                 "spark.akka.frameSize or using broadcast variables for large values."
  11.               msg = msg.format(task.taskId, task.index, serializedTask.limit, akkaFrameSize,
  12.                 AkkaUtils.reservedSizeBytes)
  13.               taskSetMgr.abort(msg)
  14.             } catch {
  15.               case e: Exception => logError("Exception in error callback", e)
  16.             }
  17.           }
  18.         }
  19.         else {
  20.           val executorData = executorDataMap(task.executorId)
  21.           executorData.freeCores -= scheduler.CPUS_PER_TASK
  22.           executorData.executorEndpoint.send(LaunchTask(new SerializableBuffer(serializedTask)))    // 獲取到executorEndpoint併發射到ExecutorBackend
  23.         }
  24.       }
  25.     }
找到CoarseGrainedExecutorBackend的receive方法：
  1. override def receive: PartialFunction[Any, Unit] = {
  2.     case RegisteredExecutor =>
  3.       logInfo("Successfully registered with driver")
  4.       val (hostname, _) = Utils.parseHostPort(hostPort)
  5.       executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)

  7.     case RegisterExecutorFailed(message) =>
  8.       logError("Slave registration failed: " + message)
  9.       System.exit(1)

  11.     case LaunchTask(data) =>                                                                            // 接收任務
  12.       if (executor == null) {
  13.         logError("Received LaunchTask command but executor was null")
  14.         System.exit(1)
  15.       } else {
  16.         val taskDesc = ser.deserialize[TaskDescription](data.value)
  17.         logInfo("Got assigned task " + taskDesc.taskId)
  18.         executor.launchTask(this, taskId = taskDesc.taskId, attemptNumber = taskDesc.attemptNumber,        // 交由Executor執行
  19.           taskDesc.name, taskDesc.serializedTask)
  20.       }

  22.     case KillTask(taskId, _, interruptThread) =>
  23.       if (executor == null) {
  24.         logError("Received KillTask command but executor was null")
  25.         System.exit(1)
  26.       } else {
  27.         executor.killTask(taskId, interruptThread)
  28.       }

  30.     case StopExecutor =>
  31.       logInfo("Driver commanded a shutdown")
  32.       executor.stop()
  33.       stop()
  34.       rpcEnv.shutdown()
  35.   }
Executor中有一個執行緒池物件，每分發一個任務就會啟一個執行緒去執行Task。



簡要總結：

DAGScheduler:
               1.handleJobSubmitted(): 根據提交過來的最後一個RDD封裝成一個ResultStage
               2.submitStage(): 根據這個ResultStage遞迴提交父RDD組成的ShuffleMapStage
               3.submitMissingTasks() : 根據Stage封裝成ShuffleMapTask或者ResultTask再封裝成TaskSet呼叫TaskScheduler提交

TaskScheduler:
               1.submitTasks() : 將TaskSet封裝成TaskSetManager提交至Executor端
                2.launchTasks() : 透過resourceOffers()方法獲取到TaskDescription集合後
                                    透過再透過該方法提交至ExecutorBackend
             


SchedulerBackend:
                  1.makeOffers() : 該方法呼叫TaskScheduler的resourceOffers(workOffers)
                                   workOffers為executorBackend註冊的子節點資訊
                  2.resourceOffers() : 透過該方法將workOffers順序打散後再從佇列中獲取到一組
                                    TaskSetManager，然後呼叫TaskSetManager的resourceOffer()
                                    方法獲取到每個節點上最合適的Task並封裝成TaskDescription


Executor :
               1.執行緒在run()中將Task反序列化解析出來呼叫run()方法
               2.run()方法中呼叫子類(ShuffleMapTask、ResultTask)的runTask()方法
               3.runTask中呼叫RDD的iterator()方法，迭代計算父RDD。
               4.在iterator()方法中如果當前RDD的storage level變數標記不是NONE的話，表示該RDD呼叫了快取方法，在BlockManager中應有儲存，那麼呼叫           
                  CacheManager中的getOrCompute()函式計算RDD














來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/29754888/viewspace-1813916/，如需轉載，請註明出處，否則將追究法律責任。

相關文章