1

在使用 apache Livy 0.7.0 创建新会话时，我遇到了错误。我还使用 zeppelin notebook（livy 解释器）来创建会话。

使用 Scala 版本 2.12.10、Java HotSpot(TM) 64 位服务器 VM、11.0.11
Spark 3.0.2 zeppelin 0.9.0

知道为什么我会收到错误吗？

我已经检查了我们在类路径中有 livy-repl_2.11-0.7.1-incubating.jar 并且 JAR 已经有它无法找到的类。

$ jar tf livy-repl_2.11-0.7.1-incubating.jar | grep "ReplDriver"
org/apache/livy/repl/ReplDriver$$anonfun$addFile$1.class
org/apache/livy/repl/ReplDriver$$anonfun$1.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$1.class
org/apache/livy/repl/ReplDriver$$anonfun$addJarOrPyFile$1.class
org/apache/livy/repl/ReplDriver.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$2.class
org/apache/livy/repl/ReplDriver$$anonfun$2.class

错误日志

    21/06/17 13:13:28 INFO LineBufferedStream: Exception in thread "main" java.lang.ClassNotFoundException: org.apache.livy.repl.ReplDriver
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)


21/06/17 13:13:28 WARN ContextLauncher: Child process exited with code 1.
21/06/17 13:13:28 ERROR RSCClient: Failed to connect to context.
java.io.IOException: Child process exited with code 1.
        at org.apache.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:397)
        at org.apache.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:448)
        at java.base/java.lang.Thread.run(Thread.java:834)
21/06/17 13:13:28 ERROR SparkProcApp: spark-submit exited with code 1
21/06/17 13:13:28 INFO RSCClient: Failing pending job f05c2bac-6e0a-4732-a419-0bde4dcfd06b due to shutdown.
21/06/17 13:13:28 INFO InteractiveSession: Stopping InteractiveSession 0...
21/06/17 13:13:28 INFO InteractiveSession: Failed to ping RSC driver for session 0. Killing application.
21/06/17 13:13:28 INFO InteractiveSession: Stopped InteractiveSession 0.
21/06/17 13:13:28 WARN InteractiveSession: Fail to get rsc uri
java.util.concurrent.ExecutionException: java.io.IOException: Child process exited with code 1.
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at org.apache.livy.server.interactive.InteractiveSession$$anonfun$18.apply(InteractiveSession.scala:411)
4

1 回答 1

0

Spark 3.0.x 带有 scala 2.12 版本。您需要使用 scal 2.12 使用 Spark 3.0.x 构建 livy 来解决此问题。然后你需要调整你的 livy.conf 这是关于如何使用 maven 重建你的 livy 的文章（如何使用 scala 2.12 重建 apache Livy）

祝你好运。我遇到了同样的问题，并且能够通过上述步骤解决。

于 2021-11-01T05:49:59.980 回答