apache-spark - Spark on Yarn 错误：Yarn 应用程序已经结束！它可能已被杀死或无法启动应用程序主控

Question

开始spark-shell --master yarn --deploy-mode client时出现错误：

纱线应用已经结束！它可能已被杀死或无法启动应用程序主控。

以下是 Yarn 的完整日志：

19/08/28 00:54:55 INFO client.RMProxy: 在 /0.0.0.0:8032 连接到 ResourceManager

容器： rhel7-cloudera-dev_33917 上的 container_1566921956926_0010_01_000001 ========================================== ======================================= LogType:stderr 日志上传时间:28-Aug-2019 00:46:31 LogLength:523 日志内容：SLF4J：类路径包含多个 SLF4J 绑定。SLF4J：在 [jar:file:/yarn/local/usercache/rhel/filecache/26/__spark_libs__5634501618166443611.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] 中找到绑定在 [jar:file:/etc/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J 中找到绑定：参见http ://www.slf4j.org/codes.html#multiple_bindings进行解释。SLF4J：实际绑定的类型为 [org.slf4j.impl.Log4jLoggerFactory]

LogType:stdout 日志上传时间：2019 年 8 月 28 日 00:46:31 LogLength:5597 日志内容：2019-08-28 00:46:19 INFO SignalUtils:54 - 为 TERM 2019-08-28 00 注册的信号处理程序： 46:19 INFO SignalUtils:54 - 为 HUP 注册的信号处理程序 2019-08-28 00:46:19 INFO SignalUtils:54 - 为 INT 注册的信号处理程序 2019-08-28 00:46:19 INFO SecurityManager:54 - 更改视图acls 为：yarn,rhel 2019-08-28 00:46:19 INFO SecurityManager:54 - 将修改 acls 更改为：yarn,rhel 2019-08-28 00:46:19 INFO SecurityManager:54 - 将视图 acls 组更改为： 2019-08-28 00:46:19 INFO SecurityManager:54 - 将修改 acls 组更改为：2019-08-28 00:46:19 INFO SecurityManager:54 - SecurityManager：身份验证已禁用；ui acls 禁用；具有查看权限的用户：Set(yarn, rhel); 具有查看权限的组：Set()；具有修改权限的用户：Set(yarn, rhel); 具有修改权限的组：Set() 2019-08-28 00:46:20 INFO ApplicationMaster:54 - 准备本地资源 2019-08-28 00:46:21 INFO ApplicationMaster:54 - ApplicationAttemptId: appattempt_1566921956926_0010_000001 2019-08-28 00 :46:21 INFO ApplicationMaster:54 - 等待 Spark 驱动程序可访问。2019-08-28 00:46:21 INFO ApplicationMaster:54 - 驱动程序现在可用：rhel7-cloudera-dev:34872 2019-08-28 00:46:21 INFO TransportClientFactory:267 - 成功创建与 rhel7-cloudera-dev 的连接/192.168.56.112:34872 107 毫秒后（0 毫秒花费在引导程序中）2019-08-28 00:46:22 INFO ApplicationMaster:54 - =================== ==================================================== ========== YARN 执行器启动上下文：env: CLASSPATH -> {{PWD}}{{PWD}}/ 瑞尔）；具有修改权限的组：Set() 2019-08-28 00:46:20 INFO ApplicationMaster:54 - 准备本地资源 2019-08-28 00:46:21 INFO ApplicationMaster:54 - ApplicationAttemptId: appattempt_1566921956926_0010_000001 2019-08-28 00 :46:21 INFO ApplicationMaster:54 - 等待 Spark 驱动程序可访问。2019-08-28 00:46:21 INFO ApplicationMaster:54 - 驱动程序现在可用：rhel7-cloudera-dev:34872 2019-08-28 00:46:21 INFO TransportClientFactory:267 - 成功创建与 rhel7-cloudera-dev 的连接/192.168.56.112:34872 107 毫秒后（0 毫秒花费在引导程序中）2019-08-28 00:46:22 INFO ApplicationMaster:54 - =================== ==================================================== ========== YARN 执行器启动上下文：env: CLASSPATH -> {{PWD}}{{PWD}}/ 瑞尔）；具有修改权限的组：Set() 2019-08-28 00:46:20 INFO ApplicationMaster:54 - 准备本地资源 2019-08-28 00:46:21 INFO ApplicationMaster:54 - ApplicationAttemptId: appattempt_1566921956926_0010_000001 2019-08-28 00 :46:21 INFO ApplicationMaster:54 - 等待 Spark 驱动程序可访问。2019-08-28 00:46:21 INFO ApplicationMaster:54 - 驱动程序现在可用：rhel7-cloudera-dev:34872 2019-08-28 00:46:21 INFO TransportClientFactory:267 - 成功创建与 rhel7-cloudera-dev 的连接/192.168.56.112:34872 107 毫秒后（0 毫秒花费在引导程序中）2019-08-28 00:46:22 INFO ApplicationMaster:54 - =================== ==================================================== ========== YARN 执行器启动上下文：env: CLASSPATH -> {{PWD}}{{PWD}}/ Set() 2019-08-28 00:46:20 INFO ApplicationMaster:54 - 准备本地资源 2019-08-28 00:46:21 INFO ApplicationMaster:54 - ApplicationAttemptId: appattempt_1566921956926_0010_000001 2019-08-28 00:46:21 INFO ApplicationMaster:54 - 等待 Spark 驱动程序可访问。2019-08-28 00:46:21 INFO ApplicationMaster:54 - 驱动程序现在可用：rhel7-cloudera-dev:34872 2019-08-28 00:46:21 INFO TransportClientFactory:267 - 成功创建与 rhel7-cloudera-dev 的连接/192.168.56.112:34872 107 毫秒后（0 毫秒花费在引导程序中）2019-08-28 00:46:22 INFO ApplicationMaster:54 - =================== ==================================================== ========== YARN 执行器启动上下文：env: CLASSPATH -> {{PWD}}{{PWD}}/ Set() 2019-08-28 00:46:20 INFO ApplicationMaster:54 - 准备本地资源 2019-08-28 00:46:21 INFO ApplicationMaster:54 - ApplicationAttemptId: appattempt_1566921956926_0010_000001 2019-08-28 00:46:21 INFO ApplicationMaster:54 - 等待 Spark 驱动程序可访问。2019-08-28 00:46:21 INFO ApplicationMaster:54 - 驱动程序现在可用：rhel7-cloudera-dev:34872 2019-08-28 00:46:21 INFO TransportClientFactory:267 - 成功创建与 rhel7-cloudera-dev 的连接/192.168.56.112:34872 107 毫秒后（0 毫秒花费在引导程序中）2019-08-28 00:46:22 INFO ApplicationMaster:54 - =================== ==================================================== ========== YARN 执行器启动上下文：env: CLASSPATH -> {{PWD}}{{PWD}}/ 54 - 准备本地资源 2019-08-28 00:46:21 INFO ApplicationMaster:54 - ApplicationAttemptId: appattempt_1566921956926_0010_000001 2019-08-28 00:46:21 INFO ApplicationMaster:54 - 等待 Spark 驱动程序可访问。2019-08-28 00:46:21 INFO ApplicationMaster:54 - 驱动程序现在可用：rhel7-cloudera-dev:34872 2019-08-28 00:46:21 INFO TransportClientFactory:267 - 成功创建与 rhel7-cloudera-dev 的连接/192.168.56.112:34872 107 毫秒后（0 毫秒花费在引导程序中）2019-08-28 00:46:22 INFO ApplicationMaster:54 - ================== ==================================================== ========== YARN 执行器启动上下文：env: CLASSPATH -> {{PWD}}{{PWD}}/ 54 - 准备本地资源 2019-08-28 00:46:21 INFO ApplicationMaster:54 - ApplicationAttemptId: appattempt_1566921956926_0010_000001 2019-08-28 00:46:21 INFO ApplicationMaster:54 - 等待 Spark 驱动程序可访问。2019-08-28 00:46:21 INFO ApplicationMaster:54 - 驱动程序现在可用：rhel7-cloudera-dev:34872 2019-08-28 00:46:21 INFO TransportClientFactory:267 - 成功创建与 rhel7-cloudera-dev 的连接/192.168.56.112:34872 107 毫秒后（0 毫秒花费在引导程序中）2019-08-28 00:46:22 INFO ApplicationMaster:54 - =================== ==================================================== ========== YARN 执行器启动上下文：env: CLASSPATH -> {{PWD}}{{PWD}}/ 21 INFO ApplicationMaster:54 - 等待 Spark 驱动程序可访问。2019-08-28 00:46:21 INFO ApplicationMaster:54 - 驱动程序现在可用：rhel7-cloudera-dev:34872 2019-08-28 00:46:21 INFO TransportClientFactory:267 - 成功创建与 rhel7-cloudera-dev 的连接/192.168.56.112:34872 107 毫秒后（0 毫秒花费在引导程序中）2019-08-28 00:46:22 INFO ApplicationMaster:54 - =================== ==================================================== ========== YARN 执行器启动上下文：env: CLASSPATH -> {{PWD}}{{PWD}}/ 21 INFO ApplicationMaster:54 - 等待 Spark 驱动程序可访问。2019-08-28 00:46:21 INFO ApplicationMaster:54 - 驱动程序现在可用：rhel7-cloudera-dev:34872 2019-08-28 00:46:21 INFO TransportClientFactory:267 - 成功创建与 rhel7-cloudera-dev 的连接/192.168.56.112:34872 107 毫秒后（0 毫秒花费在引导程序中）2019-08-28 00:46:22 INFO ApplicationMaster:54 - =================== ==================================================== ========== YARN 执行器启动上下文：env: CLASSPATH -> {{PWD}}{{PWD}}/spark_conf {{PWD}}/ spark_libs / $HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/ $HADOOP_COMMON_HOME/share/hadoop/common/lib/ $HADOOP_HDFS_HOME/share/hadoop/hdfs/ $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib / $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/ $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/ $HADOOP_YARN_HOME/share/hadoop/yarn/ $HADOOP_YARN_HOME/share/hadoop/yarn/lib/* $HADOOP_COMMON_HOME/ $HADOOP_COMMON_HOME/lib/ $HADOOP_HDFS_HOME/ $HADOOP_HDFS_HOME/lib/ $HADOOP_MAPRED_HOME/ $HADOOP_MAPRED_HOME/lib/ $HADOOP_YARN_HOME/ $HADOOP_YARN_HOME/lib/ $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/ /etc/hadoop-2.6.0/etc/hadoop:/etc/hadoop-2.6.0/share/hadoop/common/lib/ :/etc/hadoop-2.6.0 /share/hadoop/common/ :/etc/hadoop-2.6.0/share/hadoop/hdfs:/etc/hadoop-2.6.0/share/hadoop/hdfs/lib/ :/etc/hadoop-2.6.0/共享/hadoop/hdfs/:/etc/hadoop-2.6.0/share/hadoop/yarn/lib/:/etc/hadoop-2.6.0/share/hadoop/yarn/:/etc/hadoop-2.6.0/ _ _共享/hadoop/mapreduce/lib/:/etc/hadoop-2.6.0/share/hadoop/mapreduce/:/etc/hadoop-2.6.0/contrib/capacity-scheduler/.jar{{PWD}}/spark_conf/ _ _ _ _ hadoop_conf SPARK_DIST_CLASSPATH -> /etc/hadoop-2.6.0/etc/hadoop:/etc/hadoop-2.6.0/share/hadoop/common/lib/:/etc/hadoop-2.6.0/share/hadoop/common/ :/etc/hadoop-2.6.0/share/hadoop/hdfs:/etc/hadoop-2.6.0/share/hadoop/hdfs/lib/ : /etc/hadoop-2.6.0/share/hadoop/hdfs/ :/etc/hadoop-2.6.0/share/hadoop/yarn/lib/ :/etc/hadoop-2.6.0/share/hadoop/yarn/ : /etc/hadoop-2.6.0/share/hadoop/mapreduce/lib/ :/etc/hadoop-2.6.0/share/hadoop/mapreduce/ :/etc/hadoop-2.6.0/contrib/capacity-scheduler/。 jar SPARK_YARN_STAGING_DIR -> *********（已编辑）SPARK_USER -> *********（已编辑）SPARK_CONF_DIR -> /etc/spark/conf SPARK_HOME -> /etc/spark

命令：{{JAVA_HOME}}/bin/java\-server\-Xmx1024m\-Djava.io.tmpdir={{PWD}}/tmp\'-Dspark.driver.port=34872'\-Dspark.yarn.app .container.log.dir= \ -XX:OnOutOfMemoryError='kill %p' \ org.apache.spark.executor.CoarseGrainedExecutorBackend \ --driver-url \ spark://CoarseGrainedScheduler@rhel7-cloudera-dev:34872 \ - -executor-id\\--hostname\\--cores\1\--app-id\application_1566921956926_0010\--user-class-path\file:$PWD/app.jar \ 1>/stdout\2>/标准错误

资源： spark_libs -> 资源{方案：“hdfs”主机：“rhel7-cloudera-dev”端口：9000 文件：“/user/rhel/.sparkStaging/application_1566921956926_0010/ spark_libs__5634501618166443611.zip”} 大小：232107209 时间戳：1566933362存档可见性：PRIVATE __spark_conf -> 资源 { 方案：“hdfs”主机：“rhel7-cloudera-dev”端口：9000 文件：“/user/rhel/.sparkStaging/application_1566921956926_0010/ spark_conf .zip”} 大小：208377 时间戳：1566933365411类型：存档可见性：私人

==================================================== ============================== 2019-08-28 00:46:22 信息 RMProxy:98 - 在 /0.0 连接到 ResourceManager .0.0:8030 2019-08-28 00:46:22 INFO YarnRMClient:54 - 注册 ApplicationMaster 2019-08-28 00:46:22 INFO YarnAllocator:54 - 将请求 2 个执行器容器，每个容器有 1 个核心(s) 和 1408 MB 内存（包括 384 MB 开销） 2019-08-28 00:46:22 INFO YarnAllocator:54 - 提交了 2 个未本地化的容器请求。2019-08-28 00:46:22 INFO ApplicationMaster:54 - 以（心跳：3000，初始分配：200）间隔启动进度报告线程 2019-08-28 00:46:22 错误 ApplicationMaster：43 - 收到信号期限 2019 -08-28 00:46:23 INFO ApplicationMaster:54 - 最终应用状态：未定义，退出代码：16，（原因：

容器： rhel7-cloudera-dev_33917 上的 container_1566921956926_0010_02_000001 ========================================== ======================================= LogType:stderr 日志上传时间:28-Aug-2019 00:46:31 LogLength:3576 日志内容：SLF4J：类路径包含多个 SLF4J 绑定。SLF4J：在 [jar:file:/yarn/local/usercache/rhel/filecache/26/__spark_libs__5634501618166443611.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] 中找到绑定在 [jar:file:/etc/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J 中找到绑定：参见http ://www.slf4j.org/codes.html#multiple_bindings解释一下。SLF4J：实际绑定的类型为 [org.slf4j.impl.Log4jLoggerFactory] ​​线程“main”中的异常 java.io.IOException：本地异常失败：java.io.IOException；主机详情：本地主机为：“rhel7-cloudera-dev/192.168.56.112”；目标主机是：“rhel7-cloudera-dev”：9000；在 org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) 在 org.apache.hadoop.ipc.Client.call(Client.java:1474) 在 org.apache.hadoop.ipc.Client.call (Client.java:1401) 在 org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) 在 com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) 在 org.apache.hadoop。 hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) 在 sun.reflect.NativeMethodAccessorImpl。

LogType：stdout 日志上传时间：2019 年 8 月 28 日 00:46:31 LogLength：975 日志内容：2019-08-28 00:46:26 INFO SignalUtils：54 - 为 TERM 2019-08-28 00 注册的信号处理程序： 46:26 INFO SignalUtils:54 - HUP 的注册信号处理程序 2019-08-28 00:46:26 INFO SignalUtils:54 - INT 的注册信号处理程序 2019-08-28 00:46:27 INFO SecurityManager:54 - 更改视图acls 为：yarn,rhel 2019-08-28 00:46:27 INFO SecurityManager:54 - 将修改 acls 更改为：yarn,rhel 2019-08-28 00:46:27 INFO SecurityManager:54 - 将视图 acls 组更改为： 2019-08-28 00:46:27 INFO SecurityManager:54 - 将修改 acls 组更改为：2019-08-28 00:46:27 INFO SecurityManager:54 - SecurityManager：身份验证已禁用；ui acls 禁用；具有查看权限的用户：Set(yarn, rhel); 具有查看权限的组：Set()；具有修改权限的用户：Set(yarn, rhel); 具有修改权限的组：Set() 2019-08-28 00:46:28 INFO ApplicationMaster:54 - 准备本地资源 2019-08-28 00:46:28 ERROR ApplicationMaster:43 - RECEIVED SIGNAL TERM

有什么建议可以解决这个问题吗？