在我的 pyspark 程序中,我有
from pyspark import SparkConf, SparkContext, SQLContex
conf=SparkConf()
conf.setAppName("spark_name")
conf.set("spark.dynamicAllocation.enabled", "true")
conf.set("spark.shuffle.service.enabled", "true")
sc = SparkContext(conf=conf)
并将我的 pyspark 程序运行为
./spark-submit --master yarn-cluster PySparkProgramPath
然后这项工作失败了
Exception in thread "main" org.apache.spark.SparkException: Application application_1482268372614_0318 finished with failed status
在我运行yarn logs -applicationId application_1482268372614_0318
它后显示错误
ERROR ApplicationMaster: User class threw exception: java.io.IOException: Cannot run program
ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors.
yarn-client
[编辑] 当我在模式下提交时没有遇到这个错误