2

我尝试在 EMR 集群上使用 mleap，但是当我尝试使用它时，我收到以下错误：

 self._java_obj = _jvm().ml.combust.mleap.spark.SimpleSparkSerializer()
 TypeError: 'JavaPackage' object is not callable

我从代码中加载 jar 文件

开始工作的脚本：

CLUSTER_ID=XXXXX
JOB_NAME=mleap_sample
SCRIPT_PATH=s3://XXXX/mleap_model.py

aws emr --profile xxx add-steps --cluster-id $CLUSTER_ID \
--steps Name=$JOB_NAME,Jar=command-runner.jar,\
Args=[spark-submit,--deploy-mode,client,\
--conf,spark.yarn.submit.waitAppCompletion=true,\
$SCRIPT_PATH],ActionOnFailure=CONTINUE

在我的代码里面：

 sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-base_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-core_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-runtime_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-spark-base_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-spark-extension_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-tensor_2.11-0.16.0-sources.jar")

在加载期间的 EMR 日志中，我看到：

20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-base_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-base_2.11-0.16.0-sources.jar with timestamp 1594130507784
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-core_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-core_2.11-0.16.0-sources.jar with timestamp 1594130507786
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-runtime_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-runtime_2.11-0.16.0-sources.jar with timestamp 1594130507788
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-spark-base_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-spark-base_2.11-0.16.0-sources.jar with timestamp 1594130507790
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-spark-extension_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-spark-extension_2.11-0.16.0-sources.jar with timestamp 1594130507793
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-tensor_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-tensor_2.11-0.16.0-sources.jar with timestamp 1594130507796

我正在使用火花版本 2.4.5

知道为什么我会面临这个问题吗？

PS：如果我在 sagemaker 笔记本上使用 pyspark，我会收到相同的错误消息。

谢谢

4

0 回答 0