当我HiveRead.java
从 intellij ide 运行时,我可以成功运行并获得结果。然后我创建了 jar 文件(这是一个 maven 项目),然后我尝试从 IDE 运行,它给了我
ClassLoaderResolver for class "" gave error on creation : {1}
然后我查看了 SO 答案,发现我必须添加 datanulcues 罐子,我做了这样的事情
java -jar /home/saurab/sparkProjects/spark_hive/target/myJar-jar-with-dependencies.jar --jars jars/datanucleus-api-jdo-3.2.6.jar,jars/datanucleus-core-3.2.10.jar,jars/datanucleus-rdbms-3.2.9.jar,/home/saurab/hadoopec/hive/lib/mysql-connector-java-5.1.38.jar
然后我得到了这个错误
org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "datanucleus" yet this has not been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.
我发现我应该做的地方spark-submit
。所以我确实喜欢这个
./bin/spark-submit --class HiveRead --master yarn --jars jars/datanucleus-api-jdo-3.2.6.jar,jars/datanucleus-core-3.2.10.jar,jars/datanucleus-rdbms-3.2.9.jar,/home/saurab/hadoopec/hive/lib/mysql-connector-java-5.1.38.jar --files /home/saurab/hadoopec/spark/conf/hive-site.xml /home/saurab/sparkProjects/spark_hive/target/myJar-jar-with-dependencies.jar
现在我得到了新类型的错误
Table or view not found: `bigmart`.`o_sales`;
帮我 !!:)
我已将我的复制hive-site.xml
到/spark/conf
,启动 hive-metastore 服务(hiveserver2 --service metastore
)
如果有人感兴趣,这里是HiveRead.Java 代码。