0

我试图通过 jdbc 访问 spark sql，但遇到了一些错误。我正在尝试直线运行

! /usr/lib/spark/bin/beeline -u 'jdbc:hive2://host:10016/default;transportMode=binary'  -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' -e 'Error: Could not open'

host值在命令中是正确的。（特意编写以使其通用）

Error: Could not open client transport with JDBC Uri: host:10016/default;transportMode=binary: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)

hive.server2.thrift.port=10016 并且模式是二进制的。我验证了该进程正在此端口上运行。我检查了 spark thrift 服务器日志：

日志链接：https ://pastebin.com/G5Mwaw7E

它说

java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
(pasted the logs in above link).

我正在使用 spark 版本 3.1.0、hive 3.1.2 和 hadoop 3.1.2。遵循本指南 https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore

我将“spark.sql.hive.metastore.version”设置为 3.1.2 并将 jar 设置为指向 hivemetastore 3.1.2 jar，但仍然出现相同的错误。

从日志中，我还怀疑 hive 和 spark 中的 datanucleus 核心版本不匹配，但两者都使用相同的版本。
https://github.com/apache/hive/blob/branch-3.1/pom.xml#L129
https://github.com/apache/spark/blob/branch-3.1/pom.xml#L184（已编辑）

21/05/14 13:42:44 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionDriverName unknown - will be ignored
21/05/14 13:42:44 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionURL unknown - will be ignored
21/05/14 13:42:44 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionUserName unknown - will be ignored
21/05/14 13:42:44 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/05/14 13:42:44 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionPassword unknown - will be ignored
21/05/14 13:42:44 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/05/14 13:42:45 WARN Datastore: Error initialising derby schema : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' CALLED ON NULL' at line 1
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' CALLED ON NULL' at line 1
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

^ 这看起来也很可疑，metastore 是 mysql
这是一个已知问题，应该如何解决。让我知道是否需要其他任何东西。谢谢

4

0 回答 0