1

我正在尝试在 Scala 中运行 Spark 应用程序以连接到 ActiveMQ。我为此目的使用 Bahir format("org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider")。当我Bahir2.2在我built.sbt的应用程序中使用时，它运行良好，但将其更改为Bahir3.0或Bahir4.0应用程序未启动并且它给出了错误：

[error] (run-main-0) java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream`

如何解决这个问题？我可以在我的 Spark-Structured-Streaming 中使用 Bahir 的替代方法来连接到 ActiveMQ 主题吗？

编辑：我的 build.sbt

//For spark
libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "2.4.0" ,
    "org.apache.spark" %% "spark-mllib" % "2.4.0" ,
    "org.apache.spark" %% "spark-sql" % "2.4.0" ,
    "org.apache.spark" %% "spark-hive" % "2.4.0" ,
    "org.apache.spark" %% "spark-streaming" % "2.4.0" ,
    "org.apache.spark" %% "spark-graphx" % "2.4.0",
)

//Bahir
libraryDependencies += "org.apache.bahir" %% "spark-sql-streaming-mqtt" % "2.4.0"
4

1 回答 1

1

spark2.4好的，看来和之间存在某种兼容性问题bahir2.4。我通过将它们都回滚到ver 2.3.

这是我的build.sbt

name := "sparkTest"

version := "0.1"

scalaVersion := "2.11.11"

//For spark
libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "2.3.0" ,
    "org.apache.spark" %% "spark-mllib" % "2.3.0" ,
    "org.apache.spark" %% "spark-sql" % "2.3.0" ,
    "org.apache.spark" %% "spark-hive" % "2.3.0" ,
    "org.apache.spark" %% "spark-streaming" % "2.3.0" ,
    "org.apache.spark" %% "spark-graphx" % "2.3.0",
//    "org.apache.spark" %% "spark-streaming-kafka" % "1.6.3",
)

//Bahir
libraryDependencies += "org.apache.bahir" %% "spark-sql-streaming-mqtt" % "2.3.0"
于 2020-12-09T18:38:44.587 回答