按照https://georgheiler.com/2019/05/01/headless-spark-on-yarn/的教程,在 HDP 中启动自定义版本的 spark on yarn 可以正常工作,即:
# download a current headless version of spark
export SPARK_DIST_CLASSPATH=$(hadoop classpath)
export HADOOP_CONF_DIR=/usr/hdp/current/spark2-client/conf
export SPARK_HOME=<<path/to>>/spark-2.4.3-bin-without-hadoop/
<<path/to>>/spark-2.4.3-bin-without-hadoop/bin/spark-shell --master yarn --deploy-mode client --queue <<my_queue>> --conf spark.driver.extraJavaOptions='-Dhdp.version=2.6.<<version>>' --conf spark.yarn.am.extraJavaOptions='-Dhdp.version=2.6.<<version>>'
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.3
/_/
但是,一个:
spark.sql("show databases").show
只返回:
+------------+
|databaseName|
+------------+
| default|
+------------+
现在尝试传递原始的 HDP 配置(我的自定义版本的 spark 显然没有读入),例如:
一:
--files /usr/hdp/current/spark2-client/conf/hive-site.xml
二:
--conf spark.hive.metastore.uris='thrift://master001.my.corp.com:9083,thrift://master002.my.corp.com:9083,thrift://master003.my.corp.com:9083' --conf spark.hive.metastore.sasl.enabled='true' --conf hive.metastore.uris='thrift://master001.my.corp.com:9083,thrift://master002.my.corp.com:9083,thrift://master003.my.corp.com:9083' --conf hive.metastore.sasl.enabled='true'
三:
--conf spark.yarn.dist.files='/usr/hdp/current/spark2-client/conf/hive-site.xml'
四:
--conf spark.sql.warehouse.dir='/apps/hive/warehouse'
一切都无助于解决问题。如何让 spark 识别 hive 数据库?