我正在使用 AWS Glue 运行一些 pyspark python 代码,有时它成功但有时因依赖错误而失败:Resource Setup Error: Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: JohnSnowLabs#spark-nlp;2.5.4: not found]
,这是错误日志:
:: problems summary ::
:::: WARNINGS
module not found: JohnSnowLabs#spark-nlp;2.5.4
==== local-m2-cache: tried
file:/root/.m2/repository/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.pom
-- artifact JohnSnowLabs#spark-nlp;2.5.4!spark-nlp.jar:
file:/root/.m2/repository/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.jar
==== local-ivy-cache: tried
/root/.ivy2/local/JohnSnowLabs/spark-nlp/2.5.4/ivys/ivy.xml
-- artifact JohnSnowLabs#spark-nlp;2.5.4!spark-nlp.jar:
/root/.ivy2/local/JohnSnowLabs/spark-nlp/2.5.4/jars/spark-nlp.jar
==== central: tried
https://repo1.maven.org/maven2/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.pom
-- artifact JohnSnowLabs#spark-nlp;2.5.4!spark-nlp.jar:
https://repo1.maven.org/maven2/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.jar
==== spark-packages: tried
https://dl.bintray.com/spark-packages/maven/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.pom
-- artifact JohnSnowLabs#spark-nlp;2.5.4!spark-nlp.jar:
https://dl.bintray.com/spark-packages/maven/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: UNRESOLVED DEPENDENCIES ::
::::::::::::::::::::::::::::::::::::::::::::::
:: JohnSnowLabs#spark-nlp;2.5.4: not found
::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: JohnSnowLabs#spark-nlp;2.5.4: not found]
从成功运行的日志中,我可以看到胶水能够从 下载依赖项https://dl.bintray.com/spark-packages/maven/JohnSnowLabs/spark-nlp/2.5.4/spark-nlp-2.5.4.pom
,失败的作业也尝试从中下载,但失败了。
这个问题上周似乎自行解决了,但最近几天又出现了,到目前为止还没有自行解决。有没有人见过这个奇怪的问题?谢谢。