1

我正在从事从 kafka 中提取的 poc spark-streaming 工作。我可以对不安全的 kafka 0.10 集群使用相同的代码,但是当我切换到针对 ssl/kerberos ( hdp 2.5 ) 设置运行时,我遇到了一个异常:

Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:940)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:69)
at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:110)
at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
... 29 more

火花会话创建得很好,但是当执行程序启动以使用主题中的新内容时,我得到了上述异常。

提交代码相当简单:

spark-submit \
--master yarn \
--keytab ./bilsch.keytab \
--principal bilsch@HDP.SOME.ORG \
--files kafka_client_jaas.conf,bilsch.keytab \
--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.0.2 \
--repositories http://repo.hortonworks.com/content/repositories/releases \
--num-executors 1 \
--class producer \
--driver-java-options "-Djava.security.auth.login.config=./kafka_client_jaas.conf -Dhdp.version=2.5.3.0-37" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_client_jaas.conf" \
streaming_0331_2.11-1.0.jar 2>&1 | tee out

如果有帮助,我将关注michael-noll的 spark-streaming with kafka 教程

不确定需要将什么传递给我尚未通过的执行者,或者它可能只是一个 jaas 配置问题?

4

1 回答 1

0

尝试按照以下提供的步骤进行操作:-

  • 从 spark-submit 命令中删除--keytab和删除,--principal因为此信息已存在于 JAAS 配置中。
  • 运行kinit -kt bilsch.keytab bilsch@HDP.SOME.ORG,然后尝试运行 spark-submit 命令。
于 2018-01-02T16:07:59.027 回答