2

在我的 pyspark 程序中,我有

from pyspark import SparkConf, SparkContext, SQLContex  
conf=SparkConf()  
conf.setAppName("spark_name")  
conf.set("spark.dynamicAllocation.enabled", "true")  
conf.set("spark.shuffle.service.enabled", "true")  
sc = SparkContext(conf=conf)  

并将我的 pyspark 程序运行为

./spark-submit --master yarn-cluster PySparkProgramPath  

然后这项工作失败了

Exception in thread "main" org.apache.spark.SparkException: Application application_1482268372614_0318 finished with failed status

在我运行yarn logs -applicationId application_1482268372614_0318
它后显示错误

ERROR ApplicationMaster: User class threw exception: java.io.IOException: Cannot run program
ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors.

yarn-client[编辑] 当我在模式下提交时没有遇到这个错误

4

0 回答 0