0

我在我的spark配置中添加了一个包(在spark-default.conf中),但是当我使用livy创建一个新会话时,它会导致我出现问题(请参阅下面的错误)以及 session 和 death 。ps:当我删除这个包时一切正常。

20/05/04 00:17:35 WARN RSCClient: Error stopping RPC.                                                                                                                   
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@6d493840(uncancellable)                                                                      
        at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:394)                                                                               
        at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)                                                                         
   ...........
Exception in thread "Thread-32" java.io.IOException: Stream closed                                                                                                      
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)                                                                                       
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)                                                                                              
        ........                                                                                            
        at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:46)                                                                            
20/05/04 00:17:36 WARN ContextLauncher: Child process exited with code 143.                                                                                             
20/05/04 00:17:36 ERROR SparkProcApp: job was killed by user                                                                                                            
20/05/04 00:17:36 INFO InteractiveSession: Stopped InteractiveSession 0.                                                                                                
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting InteractiveSession 0 because it was inactive for more than 3600000.0 ms.                                     
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting session 0                                                                                                    
20/05/04 00:28:17 INFO InteractiveSession: Stopping InteractiveSession 0...                                                                                             
20/05/04 00:28:17 INFO InteractiveSession: Stopped InteractiveSession 0.                                                                                                
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleted session 0  

我使用:cloudera hdp2.6.5:

步 :

  • livy conf => livy.spark.master 纱线集群
  • spark-default conf => spark.jars.repositories https://dl.bintray.com/unsupervise/maven/
  • spark-defaultconf => spark.jars.packages com.github.unsupervise:spark-tss:0.1.1
4

1 回答 1

0

请尝试添加到 spark 默认值

spark.jars.repositories https://dl.bintray.com/unsupervise/maven/

或者(只是为了确保您对spark-default.conf文件没有问题)请尝试在提交 Spark 作业时将这些配置包含到 Livy 请求正文中(请参阅Livy API)。

于 2020-05-13T20:43:06.730 回答