記錄CDH Spark2的spark2-submit的一個No such file or directory問題

hackeruncle發表於2017-03-02
SparkMIT
執行:
在測試的CDH Spark2, 執行spark streaming，
命令如下：

點選(此處)摺疊或開啟

  1. spark2-submit \
  2. --class com.telenav.dataplatform.demo.realtimecases.WeatherAlerts \
  3. --master yarn --deploy-mode cluster \
  4. /usr/local/sparkProject/realtimeCases-0.0.1-SNAPSHOT.jar


錯誤:

點選(此處)摺疊或開啟

  1. 17/03/02 21:01:56 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with 1 tasks
  2. 17/03/02 21:01:56 WARN net.ScriptBasedMapping: Exception running /etc/spark2/conf.cloudera.spark2_on_yarn/yarn-conf/topology.py 172.16.102.64
  3. java.io.IOException: Cannot run program "/etc/spark2/conf.cloudera.spark2_on_yarn/yarn-conf/topology.py" (in directory "/yarn/nm/usercache/spark/appcache/application_1488459089260_0003/container_1488459089260_0003_01_000001"): error=2, No such file or directory
  4.     at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
  5.     at org.apache.hadoop.util.Shell.runCommand(Shell.java:548)
  6.     at org.apache.hadoop.util.Shell.run(Shell.java:504)
  7.     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
  8.     at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
  9.     at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
  10.     at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
  11.     at org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
  12.     at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:81)
  13.     at org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:37)
  14.     at org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:201)
  15.     at org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:182)
  16.     at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  17.     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  18.     at org.apache.spark.scheduler.TaskSetManager.org$apache$spark$scheduler$TaskSetManager$$addPendingTask(TaskSetManager.scala:182)
  19.     at org.apache.spark.scheduler.TaskSetManager$$anonfun$1.apply$mcVI$sp(TaskSetManager.scala:161)
  20.     at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
  21.     at org.apache.spark.scheduler.TaskSetManager.<init>(TaskSetManager.scala:160)
  22.     at org.apache.spark.scheduler.TaskSchedulerImpl.createTaskSetManager(TaskSchedulerImpl.scala:222)
  23.     at org.apache.spark.scheduler.TaskSchedulerImpl.submitTasks(TaskSchedulerImpl.scala:186)
  24.     at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1058)
  25.     at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:933)
  26.     at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:873)
  27.     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1632)
  28.     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1624)
  29.     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1613)
  30.     at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  31. Caused by: java.io.IOException: error=2, No such file or directory
  32.     at java.lang.UNIXProcess.forkAndExec(Native Method)
  33.     at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
  34.     at java.lang.ProcessImpl.start(ProcessImpl.java:134)
  35.     at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)

解決思路:
1.分析這句話,
17/03/02 21:01:56 WARN net.ScriptBasedMapping: Exception running /etc/spark2/conf.cloudera.spark2_on_yarn/yarn-conf/topology.py 172.16.102.64
java.io.IOException: Cannot run program "/etc/spark2/conf.cloudera.spark2_on_yarn/yarn-conf/topology.py" (in directory "/yarn/nm/usercache/spark/appcache/application_1488459089260_0003/container_1488459089260_0003_01_000001"): error=2, No such file or directory
說明在這個ip的機器上 沒有這個py檔案。

然後去機器驗證，
然後再將01機器的 配置檔案 全部copy到另外四臺即可。
scp -r /etc/spark2/conf.cloudera.spark2_on_yarn
scp -r /etc/spark2/conf.cloudera.spark2_on_yarn
scp -r /etc/spark2/conf.cloudera.spark2_on_yarn
scp -r /etc/spark2/conf.cloudera.spark2_on_yarn

驗證:
就ok了

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/30089851/viewspace-2134627/，如需轉載，請註明出處，否則將追究法律責任。

相關文章