我在我的spark配置中添加了一个包(在spark-default.conf中),但是当我使用livy创建一个新会话时,它会导致我出现问题(请参阅下面的错误)以及 session 和 death 。ps:当我删除这个包时一切正常。
20/05/04 00:17:35 WARN RSCClient: Error stopping RPC.
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@6d493840(uncancellable)
at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:394)
at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
...........
Exception in thread "Thread-32" java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
........
at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:46)
20/05/04 00:17:36 WARN ContextLauncher: Child process exited with code 143.
20/05/04 00:17:36 ERROR SparkProcApp: job was killed by user
20/05/04 00:17:36 INFO InteractiveSession: Stopped InteractiveSession 0.
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting InteractiveSession 0 because it was inactive for more than 3600000.0 ms.
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting session 0
20/05/04 00:28:17 INFO InteractiveSession: Stopping InteractiveSession 0...
20/05/04 00:28:17 INFO InteractiveSession: Stopped InteractiveSession 0.
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleted session 0
我使用:cloudera hdp2.6.5:
- 火花2.3
- 活生生的 0.7.0
- Hadoop 2.7
- lib 无监督(https://github.com/unsupervise/spark-tss)
步 :
- livy conf => livy.spark.master 纱线集群
- spark-default conf => spark.jars.repositories https://dl.bintray.com/unsupervise/maven/
- spark-defaultconf => spark.jars.packages com.github.unsupervise:spark-tss:0.1.1