scala - java.lang.NoClassDefFoundError: scala/reflect/ClassManifest

Question

尝试在 spark 上运行示例时出现错误。任何人都可以让我知道我需要对我的 pom.xml 做哪些更改才能使用 spark 运行程序。

Answer 1

目前 Spark 仅适用于 Scala 2.9.3。它不适用于更高版本的 Scala。当我尝试运行指向 2.10.2 安装的SparkPi示例时，我看到了您描述的错误。SCALA_HOME当我指向SCALA_HOME2.9.3 安装时，事情对我有用。详情在这里。

Answer 2

您应该将 scala-reflect 的依赖添加到您的 Maven 构建中：

<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-reflect</artifactId>
    <version>2.10.2</version>
</dependency>

Answer 3

Ran into the same issue using the Scala-Redis 2.9 client (incompatible with Scala 2.10) and including a dependency to scala-reflect does not help. Indeed, scala-reflect is packaged as its own jar but does not include the Class missing which is deprecated since Scala 2.10.0 (see this thread).

The correct answer is to point to an installation of Scala which includes this class (In my case using the Scala-Redis client, the answer of McNeill helped. I pointed to Scala 2.9.3 using SBT and everything worked as expected)

Answer 4

就我而言，错误是在 Kafka 的 api 中引发的。我将依赖项从

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.9.2</artifactId>
    <version>0.8.1.1</version>
</dependency>

至

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka_2.10</artifactId>
    <version>1.6.1</version>
</dependency>

解决了这个问题。