1

When using the Apache Spark EC2 script to launch a cluster I have found somewhat of a bug which is beginning to hit my pocket. When specifying the number of slaves: if you enter a number which is greater than or equal to your limit then the cluster is launched with your maximum number of slaves - but no master! This gives you no control over the slaves and thus cluster.

I have not found a way to launch just a master with the Apache Spark EC2 script. I have manually shut down 1 of the slaves to make space for a master; however, when then trying to re-launch the script it just says:

Searching for existing cluster my_cluster...
Found 0 master(s), 4 slaves
ERROR: There are already instances running in group my_cluster-master
or my_cluster-slaves

To overcome this, I have to log into the AWS console and terminate all the clusters and then restart. As Amazon charge per hour I am being charged for a full hours worth of my maximum number of clusters - all for nothing.

Is there a way to launch a master when slaves already exist?

4

1 回答 1

1

这是因为 spark-ec2 向 EC2 发出 2 个单独的请求来分配实例，一个用于主服务器，一个用于从服务器。

正如您可能猜到的那样，它在 slaves之后分配 master 实例，这导致了您所看到的问题。

当从站已经存在时，无法启动主站。只支持相反的情况——当主服务器已经存在时启动从服务器。

这种在从属服务器之后启动主服务器的行为听起来像是一个错误。

如果您想报告它以便修复它，我建议在 EC2 组件下的Apache JIRA for Spark 上创建一个问题。我会看看它。

于 2015-04-17T19:45:11.260 回答