10

I'm trying to run Spark jobs on a Dataproc cluster, but Spark will not start due to Yarn being misconfigured.

I receive the following error when running "spark-shell" from the shell (locally on the master), as well as when uploading a job through the web-GUI and the gcloud command line utility from my local machine:

15/11/08 21:27:16 ERROR org.apache.spark.SparkContext: Error initializing     SparkContext.
java.lang.IllegalArgumentException: Required executor memory (38281+2679 MB) is above the max threshold (20480 MB) of this cluster! Please increase the value of 'yarn.s
cheduler.maximum-allocation-mb'.

I tried modifying the value in /etc/hadoop/conf/yarn-site.xml but it didn't change anything. I don't think it pulls the configuration from that file.

I've tried with multiple cluster combinations, at multiple sites (mainly Europe), and I only got this to work with the low memory version (4-cores, 15 gb memory).

I.e. this is only a problem on the nodes configured for memory higher than the yarn default allows.

4

1 回答 1

10

对于您遇到的这些问题，我们深表歉意！看起来这是一个已知问题的一部分，其中某些内存设置最终是根据主计算机的大小而不是工作计算机的大小计算的，我们希望很快在即将发布的版本中解决这个问题。

目前有两种解决方法：

  1. 使用内存等于或小于工作机器类型的主机类型。

  2. --conf如果从 SSH 连接运行，则使用标志显式设置 spark.executor.memory 和 spark.executor.cores，例如：

    spark-shell --conf spark.executor.memory=4g --conf spark.executor.cores=2

    或者如果正在运行gcloud beta dataproc，请使用--properties：

    gcloud beta dataproc jobs submit spark --properties spark.executor.memory=4g,spark.executor.cores=2

您可以根据需要调整每个执行程序的核心/内存数量；在较小的执行程序方面犯错并让 YARN 将大量执行程序打包到每个工作程序上是可以的，尽管您可以通过将 spark.executor.memory 设置为每个 YARN 容器和 spark.executor 中可用的完整大小来节省每个执行程序的开销.cores 到每个工人的所有核心。

编辑：从 1 月 27 日起，新的 Dataproc 集群现在将为任何主/从机器类型的组合正确配置，如发行说明中所述。

于 2015-11-08T22:11:34.227 回答