0

我正在尝试在 Azure 云 VM 中设置 Spark 独立集群。Spark 2.2 设置已完成。如果我启动 master (start-master.sh)，我可以在 web ui 中看到 master URL。但是那个 spark master url 有那个 VM 的主机名而不是 IP 地址。VM IP 有私有和公共，我正在尝试使用公共 IP 配置主 url，以便其他客户端可以访问。

我希望如果 Spark 主机包含 IP 地址而不是主机名，那么我的应用程序可以访问主机。我尝试按照在线表格中提到的几个步骤进行操作，但没有任何效果

选项：1 我在 sbin 文件夹下创建了 spark-env.sh 并添加了 SPARK_MASTER_HOST=xxxx，但遇到了同样的问题。

选项：2

在启动 master 时，我已将主机名作为参数传递，spark-master.sh -h xxxx 但同样的错误

错误：

 your platform... using builtin-java classes where applicable
18/04/01 19:39:12 INFO SecurityManager: Changing view acls to: root
18/04/01 19:39:12 INFO SecurityManager: Changing modify acls to: root
18/04/01 19:39:12 INFO SecurityManager: Changing view acls groups to:
18/04/01 19:39:12 INFO SecurityManager: Changing modify acls groups to:
18/04/01 19:39:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7077. Attempting port 7078.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7078. Attempting port 7079.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7079. Attempting port 7080.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7080. Attempting port 7081.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7081. Attempting port 7082.
18/04/01 19:39:13 WARN Utils: Service 'sparkMaster' could not bind on port 7082. Attempting port 7083.
 Attempting port 7093.
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'sparkMaster' failed after 16 retries (starting from 7077)! Consider explicitly setting the appropriate port for the service 'sparkMaster' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:496)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:481)
@@@

如何更改火花大师网址？

4

0 回答 0