这是用于连接到我们的 hive 数据库的代码,它在一周前运行良好,但现在似乎甚至无法打开会话并获取游标来执行查询。当我明确添加 cursor.close() 方法时,该问题已暂时解决,但现在又回来了。我无法使用 python 访问 hive 数据库
我尝试使用 pyhs2 和 pyhive 两个库都无法连接到 hive 数据库。到目前为止,集群上没有任何变化。这可能是什么原因?
我知道 Hive 不是关系数据库,所以游标的概念没有意义,但是 Hive 数据库有什么方法可以记住使用 pyhive 库创建的游标吗?如果是这样,我怎样才能删除当前未使用的游标?
这是执行时引发的代码和异常
from pyhive import hive
import contextlib
class Hive():
def __init__(self,host="[hostnamehere]",db="default",port="10000",auth="KERBEROS",kerberos_service_name="hive"):
self.host = host
self.db = db
self.port = port
self.auth = auth
self.kerberos_service_name = kerberos_service_name
def connect(self):
return(hive.connect(host=self.host, port=self.port, database=self.db, auth=self.auth, kerberos_service_name=self.kerberos_service_name))
def query_one(self,sql):
with contextlib.closing(self.connect()) as connection:
with contextlib.closing(connection.cursor()) as cursor:
cursor.execute(sql)
result = cursor.fetch_one()
cursor.close()
return(result)
if __name__ == "__main__":
connector = Hive()
print("running query")
print(connector.query_one("SELECT * FROM [tablenamehere]"))
raise OperationalError(response) pyhive.exc.OperationalError: TOpenSessionResp(status=TStatus(statusCode=3, infoMessages=['*org.apache.hive.service.cli.HiveSQLException: 无法打开新会话: java.lang.RuntimeException: java.lang.RuntimeException: 无法实例化 org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient:13:12', 'org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:289 ', 'org.apache.hive.service.cli.CLIService:openSession:CLIService.java:199', 'org.apache.hive.service.cli.thrift.ThriftCLIService:getSessionHandle:ThriftCLIService.java:427', 'org .apache.hive.service.cli.thrift.ThriftCLIService:OpenSession:ThriftCLIService.java:319', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1257','org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1242', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache .thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor:process:HadoopThriftAuthBridge.java:562', 'org.apache.thrift.server.TThreadPoolServer $WorkerProcess:run:TThreadPoolServer.java:286', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624', 'java.lang.Thread:run:Thread.java:748', '*java.lang.RuntimeException:java.lang.RuntimeException: 无法实例化 org.apache.hadoop.hive.ql.metadata。SessionHiveMetaStoreClient:15:2', 'org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:547', 'org.apache.hive.service.cli.session.HiveSessionImpl:open:HiveSessionImpl. java:144', 'org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:281', '*java.lang.RuntimeException: 无法实例化 org.apache.hadoop.hive.ql。 metadata.SessionHiveMetaStoreClient:21:6', 'org.apache.hadoop.hive.metastore.MetaStoreUtils:newInstance:MetaStoreUtils.java:1566', 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient::RetryingMetaStoreClient.java:92 ', 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:getProxy:RetryingMetaStoreClient.java:138', 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:getProxy:RetryingMetaStoreClient.java:110','org.apache.hadoop.hive.ql.metadata.Hive:createMetaStoreClient:Hive.java:3510', 'org.apache.hadoop.hive.ql.metadata.Hive:getMSC:Hive.java:3542', 'org .apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:528'、'*java.lang.reflect.InvocationTargetException:null:25:4'、'sun.reflect.NativeConstructorAccessorImpl:newInstance0:NativeConstructorAccessorImpl。 java:-2'、'sun.reflect.NativeConstructorAccessorImpl:newInstance:NativeConstructorAccessorImpl.java:62'、'sun.reflect.DelegatingConstructorAccessorImpl:newInstance:DelegatingConstructorAccessorImpl.java:45'、'java.lang.reflect.Constructor:newInstance:Constructor .java:423', 'org.apache.hadoop.hive.metastore.MetaStoreUtils:newInstance:MetaStoreUtils.java:1564', '*org.apache.hadoop.hive.metastore.api.MetaException:GC 开销限制超出:30:4'、'org.apache.hadoop.hive.metastore.RetryingHMSHandler::RetryingHMSHandler.java:82'、'org.apache.hadoop.hive.metastore.RetryingHMSHandler:getProxy:RetryingHMSHandler.java: 91', 'org.apache.hadoop.hive.metastore.HiveMetaStore:newRetryingHMSHandler:HiveMetaStore.java:6463', 'org.apache.hadoop.hive.metastore.HiveMetaStoreClient::HiveMetaStoreClient.java:206', 'org.apache .hadoop.hive.ql.metadata.SessionHiveMetaStoreClient::SessionHiveMetaStoreClient.java:76', '*java.lang.OutOfMemoryError:GC 开销限制超出:0:-1'], sqlState=None, errorCode=0, errorMessage='无法打开新会话:java.lang.RuntimeException:java.lang.RuntimeException:无法实例化 org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient'),serverProtocolVersion=7,sessionHandle=无,配置=无)