hadoop&spark mapreduce對比 & 框架設計和理解

五柳-先生發表於2015-11-25
HadoopSpark框架

Hadoop MapReduce:


MapReduce在每次執行的時候都要從磁碟讀資料，計算完畢後都要把資料放到磁碟


spark map reduce:







RDD is everything for dev:


Basic Concepts:



Graph RDD:

Spark Runtime:


schedule:


Depency Type:


Scheduler Optimizations：


Event Flow:


Submit Job:


New Job Instance:


Job In Detail:


executor.launchTask:


Standalone：




Work Flow:


Standalone detail:


Driver application to Clustor:


Worker Exception:


Executor Exception:


Master Exception:


Master HA:




相關文章