1

我遇到了 Zeppelin --> Shiro --> Livy Integration 的问题。如果有人可以帮助我，那就太好了。

我目前的环境设置如下：

• 1 个主节点和 2 个从节点正在运行。
• 安装在主节点上的 Zeppelin 启动并运行
• Shiro 身份验证已使用 shiro.ini 文件启用，并且带有 shiro 的 zeppelin 也可以正常工作（还没有 LDAP 身份验证）
• 主节点上安装的 Livy 服务器启动并运行

etc/hadoop 下的 core-site.xml 配置如下：

  <property>
    <name>hadoop.proxyuser.livy.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.livy.groups</name>
    <value>*</value>
  </property>


*************************************************************
Also livy.conf under /livy/conf has been ocnfigurd as follows:
*************************************************************

# What port to start the server on.
livy.server.port                 8998

# What spark master Livy sessions should use.
livy.spark.master                yarn

livy.impersonation.enabled true


***************************************************************************
From Zeppelin UI , I have configured the %livy interpreter with below values
***************************************************************************

livy.spark.master   : local[*]
zeppelin.livy.url   : http://localhost:8998


 My Testing:
    Logged into to Zeppelin as “user1“ successfully. And to test the connectivity between the Zeppelin  - -> Shiro -- > Livy Integration I am running below simple codes .

Code 1:

%livy.spark
val accum = sc.accumulator(0) 
sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum += x)
accum.value


Output Error: Cannot start  spark.

Code 2:
%livy.spark
sc.version


Output Error on zeppelin ui: --> Cannot start  spark.

Code3:
%livy.pyspark
print("1")



Output Error: --> Cannot start spark.

Issue:
The notebook takes a while for running and throws error message "Cannot start spark." on the zeppelin UI against the notebook output. 
Further while investigating the Log file “livy-livy-server.out” under ,Log File Path: /var/log/livy/  

Below error is visible on the log file.

18/04/09 12:15:25 INFO WebServer: Starting server on http://IReplcaedmyHostNameFromHere:8999
18/04/09 12:17:56 INFO InteractiveSession$: Creating Interactive session 0: [owner: null, request: [kind: spark, proxyUser: Some(user1), conf: spark.master -> local[*], heartbeatTimeoutInSecond: 0]]
18/04/09 12:17:56 INFO RpcServer: Connected to the port 10001
18/04/09 12:17:56 WARN RSCConf: Your hostname, <My Host>, resolves to a loopback address, but we couldn't find any external IP address!
18/04/09 12:17:56 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
18/04/09 12:17:56 INFO InteractiveSessionManager: Registering new session 0
18/04/09 12:17:59 INFO LineBufferedStream: stdout: 18/04/09 12:17:59 INFO RSCDriver: Connecting to: IReplcaedmyHostNameFromHere.internal:10001
.
.
.
.
18/04/09 12:18:07 INFO LineBufferedStream: stdout: ERROR: org.apache.hadoop.security.authorize.AuthorizationException: User: livy is not allowed to impersonate user1
4

0 回答 0