我有一个 Python 脚本，它使用 pyspark 来处理一些数据，然后我试图在 AWS Glue 作业上运行它，有时作业成功但有时失败，我是新来的 glass ，这是我发现的差异之一日志中的成功运行和失败运行：
连线的事情是，在不更改任何配置或脚本的情况下，这个错误只是偶尔发生，很奇怪。
对于成功运行：
Log Contents:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/root/filecache/54/glue-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/root/filecache/19/__spark_libs__5130541944173628908.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Continuous Logging: Creating cloudwatch appender.
Continuous Logging: Creating cloudwatch appender.
21/04/13 17:13:09 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 12345@ip-123-12-123-123
21/04/13 17:13:09 INFO SignalUtils: Registered signal handler for TERM
21/04/13 17:13:09 INFO SignalUtils: Registered signal handler for HUP
21/04/13 17:13:09 INFO SignalUtils: Registered signal handler for INT
.
对于失败的运行：
Log Contents:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/root/filecache/53/glue-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/root/filecache/19/__spark_libs__1502454731345361680.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Continuous Logging: Creating cloudwatch appender.
Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.protocol.json.JsonErrorShapeMetadata.withExceptionUnmarshaller(Lcom/amazonaws/transform/JsonErrorUnmarshaller;)Lcom/amazonaws/protocol/json/JsonErrorShapeMetadata;
at com.amazonaws.services.logs.AWSLogsClient.<clinit>(AWSLogsClient.java:112)
at com.amazonaws.services.logs.AWSLogsClientBuilder.build(AWSLogsClientBuilder.java:61)
at com.amazonaws.services.logs.AWSLogsClientBuilder.build(AWSLogsClientBuilder.java:27)
at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
at com.amazonaws.services.glue.cloudwatch.CloudWatchLogsAppenderCommon.<init>(CloudWatchLogsAppenderCommon.java:86)
at com.amazonaws.services.glue.cloudwatch.CloudWatchLogsAppenderCommon$CloudWatchLogsAppenderCommonBuilder.build(CloudWatchLogsAppenderCommon.java:79)
at com.amazonaws.services.glue.cloudwatch.CloudWatchAppender.activateOptions(CloudWatchAppender.java:58)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.apache.spark.internal.Logging$class.initializeLogging(Logging.scala:120)
at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:108)
at org.apache.spark.deploy.yarn.ApplicationMaster$.initializeLogIfNecessary(ApplicationMaster.scala:785)
at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
at org.apache.spark.deploy.yarn.ApplicationMaster$.initializeLogIfNecessary(ApplicationMaster.scala:785)
at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
at org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:785)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:800)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
End of LogType:stderr
LogType:stdout
Log Upload Time:Wed Apr 14 14:33:48 +0000 2021
LogLength:0
Log Contents:
End of LogType:stdout
然后在这一行SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]之后发生了差异Continuous Logging: Creating cloudwatch appender.，然后遇到了一些错误Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.protocol.json.JsonErrorShapeMetadata.withExceptionUnmarshaller(Lcom/amazonaws/transform/JsonErrorUnmarshaller;)Lcom/amazonaws/protocol/json/JsonErrorShapeMetadata;
我不完全理解这一点，有人可以帮助我吗？谢谢。