0

I try to test Apache Spark with Hive integration on Eclipse IDE. These are the versions of each projects - Hadoop 2.7.4, Spark 2.2 and hive-2.3.2 with MySQL 5.7 on Eclipse Mars IDE. The contents of hive-site.xml is like below,

<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value> 
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>password</value>
</property>
<property>
  <name>hive.metastore.port</name>
  <value>33306</value>
</property>
<property>
    <name>hive.server2.enable.doAs</name>
    <value>false</value>
</property>

And the spark-hive Java API code on Eclipse IDE are

public static void main(String[] args) {
        Logger.getLogger("org").setLevel(Level.OFF);
        Logger.getLogger("akka").setLevel(Level.OFF);

        // TODO Auto-generated method stub
        SparkSession spark = SparkSession.builder()
                .appName("Spark Hive Test")
                .master("local[*]")
                .config("spark.sql.warehouse.dir","/user/hive/warehouse")
                .enableHiveSupport()
                .getOrCreate();

        System.out.println(spark.version());

        String tableName = "testHiveDriverTable";
      spark.sql("drop table if exists " + tableName);
      String createTable = "create table " + tableName + " (key int, value string) "
                + "ROW FORMAT DELIMITED "
                + "FIELDS TERMINATED BY ',' "
                + "LINES TERMINATED BY '\n' "
                + "STORED AS TEXTFILE";

        spark.sql(createTable);
      spark.sql("LOAD DATA LOCAL INPATH '/home/m_usr/temp/hiveTest.txt' INTO TABLE " + tableName);

      // Queries are expressed in HiveQL
      spark.sql("SELECT * FROM " + tableName).show();

      // Aggregation queries are also supported.
      spark.sql("SELECT COUNT(*) FROM " + tableName).show();


        spark.stop();
        spark.close();
    }

This code throw the exception:

java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.setReadOnly(Unknown Source)
    at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1324)
    at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:262)
    at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
    at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
Caused by: ERROR 25505: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericAuthorizer.setReadOnlyConnection(Unknown Source)
    at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.setReadOnly(Unknown Source)
    ... 8 more

But when Eclipse IDE be executed on root mode, no exceptions are thrown. I think this issue is related with privileges problem. However I have no idea which process bring this issue. And one more issue is the exception is throws from Apache Derby, not MySQL. My MySQL configuration on hive-site.xml seems to be wrong.

4

0 回答 0