0

基本问题：我可以使用 Presto 作为 Hive 元存储的基础吗？

我正在尝试在亚马逊 aws 上安装 presto 和 hive，但在创建 Hive 元存储时遇到问题。除了无法连接到 Metastore 之外，Presto 似乎已安装并正常工作。问题是当我运行 Hive 时，它​​无法创建元存储。

当我运行 Hive 时，出现以下错误：

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1449)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:63)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2661)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2680)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:425)
    ... 7 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1447)
    ... 12 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
        at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:358)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:73)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1447)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:63)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2661)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2680)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:425)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
        ... 20 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:404)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:73)
    ... 17 more

我的 conf/hive-site.xml 如下：

<property>
<name>hive.metastore.uris</name>
<value>thrift://ec2..........amazonaws.com:8080</value>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore. </description>


<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:presto://ec2...........amazonaws.com:9083/hive;databaseName=metastore_db;create=true</value>
<description>JDBC connect string for a JDBC metastore</description>


<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>presto-jdbc-0.90.jar</value>
<description>Driver class name for a JDBC metastore</description>

我可以使用 presto 创建 Metastore 还是必须使用 mySQL 或 derby？如果可以的话，我将非常感谢我能得到解决这个问题的任何帮助。

谢谢！

4

4 回答 4

1

感谢您到目前为止的回复。

@Dain Sundstrom：我尝试了您对引导操作的建议，但没有真正的快乐。

在对蜂巢结构进行了更多研究之后，我认为不可能像我尝试的那样使用 presto。根据我现在的理解，presto 使用 hive 作为其数据库，但 hive 本身需要一个 mySQL 或 derby 的底层数据库，这是我出错的地方。所以我的计划是现在安装 mySQL，将 hive 链接到它，然后这样做。该计划的第 2 步是祈祷它没有太多麻烦。

谢谢你的帮助。

于 2015-02-19T09:24:10.773 回答
0

EMR有一个Presto 引导操作，它应该为您处理所有设置。

于 2015-02-17T19:25:23.693 回答
0

将 Metastore 存储到 MySQL 时，您可以使用以下步骤：

CREATE DATABASE metastore_db;

CREATE USER 'hiveuser'@'%' IDENTIFIED BY 'hivepassword';

GRANT all on *.* to 'hiveuser'@localhost identified by 'hivepassword';

flush privileges;

完成后，将以下内容添加到 $HIVE_HOME/conf/ 中的 hive-site.xml

<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://10.0.0.96/metastore_db?createDatabaseIfNotExist=true</value>
        <description>metadata is stored in a MySQL server</description>
        </property>

        <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>MySQL JDBC driver class</description>
        </property>

        <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hiveuser</value>
        <description>user name for connecting to mysql server </description>
        </property>

        <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>hivepassword</value>
        <description>password for connecting to mysql server </description>
        </property>

        <property>
        <name>hive.metastore.uris</name>
        <value>thrift://10.0.0.96:9083</value>
        <description>Hive metastore Thrift server</description>
        </property>

然后启动 metastore 服务：

蜂巢--服务元存储 &

如果您想了解更多信息或遇到问题，请发表评论。

问候，德维普拉萨德。

于 2015-02-23T14:15:02.337 回答
0

理论上你应该能够做到这一点。但是 Presto 需要一个后端；哪个支持 Presto 用于您的 Metastore 数据库？无论您使用哪种后端，它都需要支持插入。

于 2015-02-17T18:20:15.967 回答