1

, 我已经安装了
1) scala-2.10.3

2) spark-1.0.0 使用以下变量更改了 spark-env.sh

export SCALA_HOME=$HOME/scala-2.10.3

export SPARK_WORKER_MEMORY=16g

我可以看到 Spark 大师。

3) 鲨鱼-0.9.1-bin-hadoop1

Changed shark-env.sh with below variables

export SHARK_MASTER_MEM=1g

SPARK_JAVA_OPTS=" -Dspark.local.dir=/tmp "

SPARK_JAVA_OPTS+="-Dspark.kryoserializer.buffer.mb=10 "

SPARK_JAVA_OPTS+="-verbose:gc -XX:-PrintGCDetails -XX:+PrintGCTimeStamps "

    export SPARK_JAVA_OPTS

    export HIVE_HOME=/usr/share/dse/hive

export HIVE_CONF_DIR="/etc/dse/hive"

export SPARK_HOME=/home/ubuntu/spark-1.0.0

export SPARK_MEM=16g

source $SPARK_HOME/conf/spark-env.sh

4) 在 DSE 中，Hive 版本是 Hive 0.11 Existing Hive-site.xml 是

    <configuration>

     <!-- Hive Execution Parameters -->
         <property>
                <name>hive.exec.mode.local.auto</name>
                <value>false</value>
             <description>Let hive determine whether to run in local mode automatically</description>
        </property>

        <property>
                <name>hive.metastore.warehouse.dir</name>
              <value>cfs:///user/hive/warehouse</value>
             <description>location of default database for the warehouse</description>
        </property>

        <property>
             <name>hive.hwi.war.file</name>
              <value>lib/hive-hwi.war</value>
                <description>This sets the path to the HWI war file, relative to ${HIVE_HOME}</description>
          </property>

          <property>
                <name>hive.metastore.rawstore.impl</name>
                <value>com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStore</value>
                <description>Use the Apache Cassandra Hive RawStore implementation</description>
          </property>

         <property>
                <name>hadoop.bin.path</name>
                <value>${dse.bin}/dse hadoop</value>
          </property>

          <!-- Set this to true to enable auto-creation of Cassandra keyspaces as Hive Databases -->
          <property>
                <name>cassandra.autoCreateHiveSchema</name>
                <value>true</value>
          </property>
    </configuration>

5) 运行 Shark shell 时出现错误：

 Unable to instantiate Org.apache.hadoop.hive.metastore.HiveMetaStoreClient

和

6) 在使用 -skipRddReload 运行 Shark shell 时 - 我能够获取 Shark shell 但无法连接配置单元并且无法执行任何命令。

 shark> DESCRIVE mykeyspace;

并收到错误消息：

FAILED: Error in metastore: java.lang.RuntimeException: Unable to instantiate org.apache.haddop.hive.metastore.HiveMataStoreClient.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.q1.exec.DDLTask.

请详细说明如何在 Datastax 企业 (Cassandra) 上配置 spark/shark。

4

0 回答 0