1

下面的代码构建为 jar 并通过 putty 使用 sparkSubmit 命令执行。它工作正常。

var conf = new SparkConf().setAppName("ABC")

val sc = new SparkContext(conf)

var hiveContext = new HiveContext(sc)

import sqlContext.implicits._ 

sqlContext.sql("query")

但是当我通过 SparkLauncher 运行相同的代码时，它会在下面引发错误，Master - Yarn-Cluster Spark - 1.6

java.io.IOException: Cannot run program "/etc/spark/conf.cloudera.CD-SPARK_ON_YARN-brkvSOzr/yarn-conf/topology.py" (in directory "/data4/yarn/nm/usercache/ppmingusrdev/appcache/application_14823231312_123/container_14866508534534_144_01_000004"): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:548)
    at org.apache.hadoop.util.Shell.run(Shell.java:504)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
    at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
    at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
    at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
    at org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
    at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:81)
    at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$handleAllocatedContainers$2.apply(YarnAllocator.scala:337)
    at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$handleAllocatedContainers$2.apply(YarnAllocator.scala:336)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.deploy.yarn.YarnAllocator.handleAllocatedContainers(YarnAllocator.scala:336)
    at org.apache.spark.deploy.yarn.YarnAllocator.allocateResources(YarnAllocator.scala:236)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:368)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
4

1 回答 1

1

在最近将 Cloudera Manager 升级到 5.10 之后，我们今天正在处理同样的错误。在我们的例子中，这是因为这个版本的 CM 中的一个错误。

这意味着运行您的 Application Master 的工作节点（如果处于 yarn-client 模式，则不是运行驱动程序的边缘节点）没有 Spark 网关角色，因此没有 spark-on-yarn conf 目录。

我们的解决方法是为每个节点分配 Spark 网关角色并重新部署客户端配置。

顺便说一句，您的工作仍应运行，但数据局部性减少或没有（因此可能慢得多）。

于 2017-02-21T23:07:22.330 回答