1

我是 Apache spark 的新手，并试图在带有 Mobius 的 Windows 机器上运行它。但我在运行它时遇到异常。首先我设置了以下环境变量

JAVA_HOME C:\Program Files\Java\jdk1.8.0_152
HADOOP_HOME C:\winutils
SPARK_HOME  c:\spark [spark-2.0.2-bin-hadoop2.6]
SPARKCLR_HOME E:\SparkClr\spark-clr_2.11-2.0.200\runtime

在代码中提供了正确的文件以从 file:///E:/​​Mobius/Mobius/SampleText.txt 读取数据

我试图在调试模式下运行它

E:\SparkClr\spark-clr_2.11-2.0.200\runtime\scripts>sparkclr-submit.cmd debug
[sparkclr-submit.cmd] SPARKCLR_JAR=spark-clr_2.11-2.0.200.jar
[sparkclr-submit.cmd] LAUNCH_CLASSPATH="E:\SparkClr\spark-clr_2.11-2.0.200\runtime\lib\spark-clr_2.11-2.0.200.jar;c:\spark\jars\*"
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/12/19 11:34:23 INFO CSharpRunner: Starting CSharpBackend!
17/12/19 11:34:25 INFO CSharpRunner: Port number used by CSharpBackend is 5567
***********************************************************************
* [CSharpRunner.main] Backend running debug mode. Press enter to exit *
***********************************************************************
17/12/19 11:35:09 INFO SparkContext: Running Spark version 2.0.2

在我的 App.config 中添加了以下条目 [我从上面得到的端口名称]

<add key="CSharpBackendPortNumber" value="5567"/>

我收到以下异常消息

    JVM method execution failed: Nonstatic method partitions failed for class 12 when called with no parameters

Stack Trace
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters)
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName)
   at Microsoft.Spark.CSharp.Proxy.Ipc.RDDIpcProxy.GetNumPartitions()
   at Microsoft.Spark.CSharp.Core.RDD`1.GetDefaultPartitionNum()
   at Microsoft.Spark.CSharp.Core.PairRDDFunctions.PartitionBy[K,V](RDD`1 self, Int32 numPartitions, Func`2 partitionFunc)
   at Microsoft.Spark.CSharp.Core.PairRDDFunctions.ReduceByKey[K,V](RDD`1 self, Func`3 reduceFunc, Int32 numPartitions)
   at Microsoft.Spark.CSharp.Examples.WordCountExample.Main(String[] args) in E:\Mobius\Mobius\examples\Batch\WordCount\Program.cs:line 35

我不确定 CSharpWorkerPath。

如果有人更早遇到此类问题，请提供帮助。

4

0 回答 0