0

问题：
在kerberos环境下spark thrift server如何连接hiveserver2？
补充：
- spark on yarn 可以正常提交任务，比如计算pi。
- 所有配置文件都是从启用了 kerberos 的普通集群复制的

日志：

Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:113)
    at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:225)
    at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:137)
    at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:127)
    at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:58)
    at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:93)
    at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
....

命令：

bash start-thriftserver.sh \
    --name SparkSqlJDBC \
    --master yarn \
    --deploy-mode client \
    --num-executors 6 \
    --executor-memory 50g \
    --executor-cores 16 \
    --driver-memory 4g \
    --conf spark.dynamicAllocation.enabled=false \
    --hiveconf hive.server2.thrift.port=32132 \
    --hiveconf hive.server2.authentication.kerberos.principal="test_user2/test02@XXX.COM" \
    --hiveconf hive.server2.authentication.kerberos.keytab="~/test_user2.keytab" \
    --hiveconf hive.server2.thrift.bind.host=xx.xx.xxx.xxx \
    --hiveconf hive.server2.logging.operation.enabled=true \
    --hiveconf hive.server2.enable.doAs=true
4

0 回答 0