7

环境：Spark 2.3.0、Scala 2.11.12、Kafka（无论最新版本是什么）

我有一个安全的 Kafka 系统，我正在尝试将我的 Spark Streaming Consumer 连接到该系统。下面是我的build.sbt文件：

name := "kafka-streaming"
version := "1.0"

scalaVersion := "2.11.12"

// still want to be able to run in sbt
// https://github.com/sbt/sbt-assembly#-provided-configuration
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))

fork in run := true
javaOptions in run ++= Seq(
    "-Dlog4j.debug=true",
    "-Dlog4j.configuration=log4j.properties")

assemblyMergeStrategy in assembly := {
    case "META-INF/services/org.apache.spark.sql.sources.DataSourceRegister" => MergeStrategy.concat
    case PathList("META-INF", _*) => MergeStrategy.discard
    case _ => MergeStrategy.first
}

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "2.3.0",
    "org.apache.spark" %% "spark-sql" % "2.3.0",
    "org.apache.spark" %% "spark-streaming" % "2.3.0",
    "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0",
    "org.apache.spark" %% "spark-sql-kafka-0-10" % "2.3.0",
    "com.ibm.db2.jcc" % "db2jcc" % "db2jcc4"
)

请注意，这是 Spark 2.3.0，我无法更改我的 Spark 版本。

现在这是我尝试将 Spark Streaming 消费者连接到我的安全 Kafka 的部分代码：

val df = spark.readStream
            .format("kafka")
            .option("subscribe", "raw_weather")
            .option("kafka.bootstrap.servers", "<url:port>s")
            .option("kafka.security.protocol", "SASL_SSL")
            .option("kafka.sasl.mechanism" , "PLAIN")
            .option("kafka.sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"user\" password=\"" + "password" + "\";")
            .option("kafka.ssl.protocol", "TLSv1.2")
            .option("kafka.ssl.enabled.protocols", "TLSv1.2")
            .option("kafka.ssl.endpoint.identification.algorithm", "HTTPS")
            .load()

当我尝试运行这个程序时，会抛出以下错误：

Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:702)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:557)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:540)
    at org.apache.spark.sql.kafka010.SubscribeStrategy.createConsumer(ConsumerStrategy.scala:62)
    at org.apache.spark.sql.kafka010.KafkaOffsetReader.createConsumer(KafkaOffsetReader.scala:314)
    at org.apache.spark.sql.kafka010.KafkaOffsetReader.<init>(KafkaOffsetReader.scala:78)
    at org.apache.spark.sql.kafka010.KafkaSourceProvider.createContinuousReader(KafkaSourceProvider.scala:130)
    at org.apache.spark.sql.kafka010.KafkaSourceProvider.createContinuousReader(KafkaSourceProvider.scala:43)
    at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:185)
    >> at com.ibm.kafkasparkintegration.executables.WeatherDataStream$.getRawDataFrame(WeatherDataStream.scala:74)
    at com.ibm.kafkasparkintegration.executables.WeatherDataStream$.main(WeatherDataStream.scala:24)
    at com.ibm.kafkasparkintegration.executables.WeatherDataStream.main(WeatherDataStream.scala)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: unable to find LoginModule class:  org.apache.kafka.common.security.plain.PlainLoginModule
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:623)
    ... 11 more
Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class:  org.apache.kafka.common.security.plain.PlainLoginModule
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:794)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:69)
    at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
    at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
    ... 14 more

错误日志中的>>指向load()上面的代码片段。这几天我一直在尝试解决这个问题，但没有取得太大的成功。

4

0 回答 0