1

我正在使用 spark 版本 2.2.1 和 Carbondata 1.5.3 版本。按照Carbondata 官方指南中的说明,我可以运行导入语句 import org.apache.spark.sql.SparkSession import org.apache.spark.sql.CarbonSession._

但是在下面的日志中无法执行下一步:

scala> val carbon = SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession("<carbon_store_path>")
19/05/08 12:14:17 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect.
19/05/08 12:14:17 WARN CarbonProperties: The enable unsafe sort value "null" is invalid. Using the default value "true
19/05/08 12:14:17 WARN CarbonProperties: The enable off heap sort value "null" is invalid. Using the default value "true
19/05/08 12:14:17 WARN CarbonProperties: The custom block distribution value "null" is invalid. Using the default value "false
19/05/08 12:14:17 WARN CarbonProperties: The enable vector reader value "null" is invalid. Using the default value "true
19/05/08 12:14:17 WARN CarbonProperties: The carbon task distribution value "null" is invalid. Using the default value "block
19/05/08 12:14:17 WARN CarbonProperties: The enable auto handoff value "null" is invalid. Using the default value "true
19/05/08 12:14:17 WARN CarbonProperties: The specified value for property 512is invalid.
19/05/08 12:14:17 WARN CarbonProperties: The specified value for property carbon.sort.storage.inmemory.size.inmbis invalid. Taking the default value.512
java.lang.ClassNotFoundException: org.apache.spark.sql.hive.CarbonSessionStateBuilder
  at scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:348)
  at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
  at org.apache.spark.util.CarbonReflectionUtils$.createObject(CarbonReflectionUtils.scala:324)
  at org.apache.spark.util.CarbonReflectionUtils$.getSessionState(CarbonReflectionUtils.scala:220)
  at org.apache.spark.sql.CarbonSession.sessionState$lzycompute(CarbonSession.scala:57)
  at org.apache.spark.sql.CarbonSession.sessionState(CarbonSession.scala:56)
  at org.apache.spark.sql.CarbonSession$CarbonBuilder$$anonfun$getOrCreateCarbonSession$2.apply(CarbonSession.scala:260)
  at org.apache.spark.sql.CarbonSession$CarbonBuilder$$anonfun$getOrCreateCarbonSession$2.apply(CarbonSession.scala:260)
  at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
  at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
  at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
  at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
  at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
  at org.apache.spark.sql.CarbonSession$CarbonBuilder.getOrCreateCarbonSession(CarbonSession.scala:260)
  at org.apache.spark.sql.CarbonSession$CarbonBuilder.getOrCreateCarbonSession(CarbonSession.scala:169)
  ... 50 elided

我的 JAVA_HOME 已设置,否则 spark 无法运行,我可以在没有 CarbonData 的情况下运行 Spark 应用程序。尝试创建 carbonsession 时失败。SPARK_HOME 指向 Spark 的 bin 目录。

我在本地机器上运行 Spark 并使用本地文件系统进行存储而没有配置单元。

任何帮助表示赞赏。如果需要任何其他详细信息,请告诉我。

4

1 回答 1

0

确保您的 carbondata jar 适合您的 spark 版本。起初我犯了那个错误

于 2019-12-10T08:37:31.963 回答