1

我正在运行一个简单的 spark-submit 作业，例如：

enter code here spark-submit --class com.x.y.z.logan 
 /home/test/spark/sample.jar

  table in jar file

   hiveContext.sql("CREATE TABLE IF NOT EXISTS 
   databasename.tablename(es_column_name STRING) STORED BY 
   'org.elasticsearch.hadoop.hive.EsStorageHandler' 
    TBLPROPERTIES('es.resource' = 'index_name/log','es.mapping.names' 
    ='tablecoulmname :es_column_name ', 'es.nodes' = 
     '192.168.x.1y:9200','es.input.json' = 'false', 
      'es.index.read.missing.as.empty' = 'yes' ,'es.index.auto.create' = 
     'yes') ")

      hiveContext.sql("INSERT INTO TABLE test.incompleterun SELECT 
      s.streamname FROM incomplete s");



       **ERROR**
      client token: N/A
       diagnostics: User class threw exception: java.lang.RuntimeException: 
      java.lang.RuntimeException: class 
      org.elasticsearch.hadoop.mr.EsOutputFormat$EsOutputCommitter not 
      org.apache.hadoop.mapred.OutputCommitter
      ApplicationMaster host: 192.168.x.y
      ApplicationMaster RPC port: 0
      queue: root.users.test
      start time: 1485286033939
      final status: FAILED
      tracking URL: some URL
      user: test
       Exception in thread "main" org.apache.spark.SparkException: 
      Application application_1485258812942_0008 finished with failed status
      at org.apache.spark.deploy.yarn.Client.run(Client.scala:1035)
      at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1082)
      at org.apache.spark.deploy.yarn.Client.main(Client.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

当我们在 Hive 界面中创建外部表并将数据从 hive 表加载到外部表中时，ES-hadoop 工作正常。当我们在 jar 中包含相同的查询时，它不起作用。当我们创建普通的 hive 表时，相同的 jar 文件可以正常工作。当我们在 jar 文件中包含外部表时，这里的问题显示以下错误有人可以帮我解决这个问题吗？

4

0 回答 0