Spark 加入

Spark on  Yarn

client

Cluster

本質區別，driver位置不同

1)有哪些不同得程式？

2)分別有什麼作用？

3)Spark作業執行流程是什麼樣的

--master yarn

CoarseGrainedExecutorBackend 預設executor有兩個

CoarseGrainedExecutorBackend 

SparkSubmit

ApplicationMaster

跑Client有

--master client

CoarseGrainedExecutorBackend 預設executor有兩個

CoarseGrainedExecutorBackend 

SparkSubmit

ExecutorLauncher

Spark 原始碼流程

SparkSubmit.main{

呼叫： doSubmit

val submit = new SparkSubmit()

submit.doSubmit(args) {

val appArgs = parseArguments(args) {

mergeDefalutSparkProperties()   //  使用default得引數

ignoreNonSparkProperties()

loadEnvironmentArguments(){

action = Option(action).getOrElse(SUBMIT)

}

}

submit(appArgs, uninitLog){

val (childArgs, childClasspath, sparkConf, childMainClass) = prepareSubmitElements{

if(deployMode == CLIENT){ // 如果選用client模式

childMainClass = args.mainClass  // 使用者指定 類 

}

if(deployMode == CLUSTER){ // 如果選用cluster模式

childMainClass = org.apache.spark.deploy.yarn.YarnClusterApplication  // 固定類

}

}

var  mainClass == Utils.classForName(childMainClass) // 使用主main

val app : SparkApplication

app.start(childArgs.toArray, SparkConf){

new Client(new ClientArguments(args),conf).run(){

this.appId = submitApplication() // 提交應用程式， log(： requesting a new application from cluster with 1 nodeManagers)   ==== 》  獲取yarn得連結狀態，

val containerContext = createContainerLaunchContext(newAppResponse)  // client.scala

{

val amClass = 

if (isClusterMode) {

Utils.classForName("org.apache.spark,deploy.yarn.applicationMaster").getName  // cluset模式是applicationmaster

}

else {

Utils.classForName("org.apache.spark.deploy.yarn.ExecutorLauncher").getName  // client是executorLauncher

}

}

val appcontext = createApplicationSubmissionContext(newApp, containerContext)  // client.scala

}

}

}

}

SparkSubmitArgument ==> 就Submit提交解析得引數包裝

applicationMaster：

ApplicationMaster.main{

master = new Applicationmaster(am)

master.run()

runImpl(){

if(isClusterMode){

runDriver(){

userClassTread = strartUserApplication(){

val mainMethod = userClassLoader.loadClass(args.userClass).getMethod("main",classOf[Array[String]])

mainMethod.invoke(null, userArgs.toArray)

}

val sc = ThreadUtils.awaitResult(SparkContextPromis.function)

registerAm(host,port,userConf,sc.ui.map(_.webUrl))

createAllocator(driverRef, userConf){

allocator.allocateResources(){

handleAllocatoedContainers(allocatedContainers.asScala){

runAllocatedContainers(containersToUse){

//啟動container

for(container <- containersToUse){  // 兩個container 啟動

launcherPool.execute(Thread).run(){

val commands = prepareCommand(){

org.apache.spark.executor.CoarseGrainedExecutorBackend

}

nmClient.startContainer(container.get, ctx)

}

}

}

}

}else{

runExecutorLauncher()

}

}

}

推薦閱讀：

https://blog.csdn.net/weixin_43866666/article/details/121743559