1

I have installed zeppelin. Everything is working except when i try to import an hive context.

MY configuration on Zeppelin :

System.getenv().get("MASTER")
System.getenv().get("SPARK_YARN_JAR")
System.getenv().get("HADOOP_CONF_DIR")
System.getenv().get("JAVA_HOME")
System.getenv().get("SPARK_HOME")
System.getenv().get("PYSPARK_PYTHON")
System.getenv().get("PYTHONPATH")
System.getenv().get("ZEPPELIN_JAVA_OPTS")

res0: String = yarn-client
res1: String = /home/centos/zeppelin-R-rscala/interpreter/spark/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar
res2: String = /etc/hadoop/conf
res3: String = /usr/jdk64/jdk1.8.0_60
res4: String = /usr/hdp/2.3.4.0-3485/spark
res5: String = null
res6: String = /usr/hdp/current/spark-client/python/lib/py4j-0.8.2.1-src.zip:/usr/hdp/current/spark-client/python/:
res7: String = -Dhdp.version=2.3.4.0-3485

What i try to do 

%spark
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

The error i have :

 java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:529)
    at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:193)
    at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:164)
    at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:162)
    at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:415)

Note that everything works fine if i launch directly from the shell.

Thanks

4

1 回答 1

2

这可能意味着您将 Tez 作为 Hive 的执行引擎。您应该在用于 Spark 的 hive-site.xml 中更改该属性：

<property>
  <name>hive.execution.engine</name>
  <value>mr</value>
</property>

就我而言，我必须通过 Ambari 更改它，但这取决于您的设置。

于 2016-01-19T14:09:43.440 回答