1

我正在使用Spark 2.3.1和Connector/J 5.1.47。

我写了一个简单的程序来检查 Metastore 的连接性：

from pyspark.context import SparkContext
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf

conf = SparkConf()
conf.set("javax.jdo.option.ConnectionURL", "jdbc:mysql://localhost/my_metastore?createDatabaseIfNotExist=true&useSSL=false")
conf.set("javax.jdo.option.ConnectionDriverName", "com.mysql.jdbc.Driver")
conf.set("javax.jdo.option.ConnectionUserName", "root")
conf.set("javax.jdo.option.ConnectionPassword", "****")

spark = SparkSession.builder \
                .config(conf=conf) \
                .enableHiveSupport() \
                .getOrCreate()

spark.sql("SELECT NOW()").collect()

spark.stop()

令我惊讶的是，我发现停止 spark 会话后，metastore 连接仍然处于活动状态！

mysql> show processlist;
+------+------+-----------------+--------------+---------+------+----------+------------------+
| Id   | User | Host            | db           | Command | Time | State    | Info             |
+------+------+-----------------+--------------+---------+------+----------+------------------+
|    3 | lc   | localhost       | NULL         | Query   |    0 | starting | show processlist |
| 4342 | root | localhost:54368 | my_metastore | Sleep   |    5 |          | NULL             |
| 4343 | root | localhost:54369 | my_metastore | Sleep   |    5 |          | NULL             |
| 4346 | root | localhost:54372 | my_metastore | Sleep   |    5 |          | NULL             |
| 4347 | root | localhost:54373 | my_metastore | Sleep   |    5 |          | NULL             |
+------+------+-----------------+--------------+---------+------+----------+------------------+
5 rows in set (0.00 sec)

知道这是 Spark 还是 Connector/J 的错误吗？

4

0 回答 0