1

我在 Apache Spark 1.6.0 中有一个超过 3M 用户和 30k 个项目的隐式 MatrixFactorizationModel。现在我想为所有用户计算每个用户推荐的前 10 个项目，代码如下：

val model = MatrixFactorizationModel.load(sc, "/hdfs/path/to/model")
model.userFeatures.cache
model.productFeatures.cache
val recommendations: RDD[(Int, Array[Rating])] = model.recommendProductsForUsers(10)

不幸的是，这会使计算崩溃并出现以下错误：

WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_xxx_xxxxxxxxx_xxxx_xx_xxxxx on host: xxxxx.xxxx.xxx. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_xxx_xxxxx_xxxx_xx_xxxxxxx
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
        at org.apache.hadoop.util.Shell.run(Shell.java:487)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

原因是什么？如何计算推荐值？

Spark 在 54 个节点的集群上运行，我使用以下命令启动 REPL：

spark-shell  \
--master yarn \
--driver-memory 16g \
--executor-memory 16G \
--num-executors 32 \
--executor-cores 8

用户和项目因素都在具有 504 个分区的缓存 RDD 中。

4

0 回答 0