1

我正在尝试运行 Mobius HiveDataFrame Example 但无法运行。

        var jsonFilePath = "file:///C:/Mobius/build/runtime/data/people.json";
        const string dbName = "SampleHiveDataBaseForMobius";
        const string tableName = "people";

        var builder = SparkSession.Builder().EnableHiveSupport();
        // The following setting is required to use Spark 2.0 in Windows
        // It may be provided in command line when running Mobius app
        //builder = builder.Config("spark.sql.warehouse.dir", "<hdfs or local path>");
        var session = builder.GetOrCreate();
        var peopleDataFrame = session.Read().Json(jsonFilePath);
        session.Sql(string.Format("CREATE DATABASE IF NOT EXISTS {0}", dbName)); // create database if not exists
        session.Sql(string.Format("USE {0}", dbName));
        //hiveContext.Sql(string.Format("DROP TABLE {0}", tableName)); // drop table if exists

        peopleDataFrame.Write().Mode(SaveMode.Overwrite).SaveAsTable(tableName); // create table
        var tablesDataFrame = session.Table(tableName); // get all tables in database
        logger.LogInfo(string.Format("table count in database {0}: {1}", dbName, tablesDataFrame.Count()));
        tablesDataFrame.Show();

        session.Sql(string.Format("SELECT * FROM {0}", tableName)).Show(); // select from table

我收到错误@

var peopleDataFrame = session.Read().Json(jsonFilePath);

来源：Microsoft.Spark.CSharp.Adapter

消息：JVM 方法执行失败：在没有参数的情况下调用类 14 的非静态方法加载失败

这是堆栈跟踪

Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters)
Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName)
Microsoft.Spark.CSharp.Proxy.Ipc.DataFrameReaderIpcProxy.Load()
Microsoft.Spark.CSharp.Sql.DataFrameReader.Load()
Microsoft.Spark.CSharp.Sql.DataFrameReader.Load(String path)
Microsoft.Spark.CSharp.Sql.DataFrameReader.Json(String path)
DemoSparkHiveDataFrame.Program.Main(String[] args) in e:\Work\Feb\VS 2012\DemoSparkHiveDataFrame\DemoSparkHiveDataFrame\Program.cs:line 26
System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
System.Threading.ThreadHelper.ThreadStart_Context(Object state)
System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
System.Threading.ThreadHelper.ThreadStart()
4

1 回答 1

1

我有一个解决方案，我只需要设置

builder.config("spark.sql.warehouse.dir","file:///C:/spark-warehouse")

设置 spark.sql.warehouse.dir 路径，它对我来说很好。

于 2017-03-27T11:19:29.400 回答